---
title: "ENABLE Delphi study analysis"
date: ' `r Sys.Date()`'
output:
  html_document: default
  tables: yes
  theme: flatly
  toc: yes
  highlight: textmate
  word_document: default
  pdf_document: default
Author: ADima
---

This report presents the analysis of the ENABLE Delphi survey conducted between October 2021 and January 2022 within the Working Group 2 of the ENABLE COST action. The survey aimed to collect stakeholders feedback on the proposed structure for an online repository of medication adherence technologies.
The report includes the following analyses: 

1) **Descriptive statistics** were performed to characterize the sample (and each stakeholder subgroup) regarding profession, years of experience, age, gender and country. 

2) The **level of agreement** on relevance, clarity and completeness was described based on several criteria representing different levels of agreement and consequently carrying different weights in these decisions.
- the Interpercentile Range Adjusted for Symmetry (IPRAS), an analysis technique from the RAND-UCLA Appropriateness Method (RAM).
- the disagreement index (DI) was calculated as a ratio between the Interpercentile Range (IPR) and IPRAS. A DI above 1 (i.e., IPR above IPRAS) indicates that disagreement exists. The IPR was calculated using the 30th to 70th percentile. IPRAS for the 9-points Likert scale was calculated according to the formula presented in the RAM User Manual.
- the median and DI defined different levels of agreement and steer the decisions about the repository structure. 
For relevance ratings:
- items with the median of 7-9 and no disagreement were considered as relevant and mandatory.
- items with the median of 4-6 or disagreement were considered as optional.
- items with the median of 1-3 and no disagreement, were considered not relevant and candidates for exclusion.
Median ratings were rounded to the nearest integers (e.g., 6.5 or 3.5 or above were assigned to 7 or 4). 
Stakeholders’ responses per question were summarized using descriptive statistics.
For clarity ratings, the above criteria were applied as (i) sufficiently clear to remain unchanged; (ii) optional changes and (iii) candidates for rephrasing.
Panellist comments in the open text fields were analysed qualitatively in a separate document, using content analysis.

2) **Subgroup analyses** per stakeholder group examined variation in opinions and potential differences among subgroups. The same agreement criteria were applied and descriptive statistics were stratified by stakeholder group. 
The reliability of ratings per question within stakeholder group was reported by calculating the intraclass correlation coefficient (ICC). The ICC calculation was based on the two-way random model, considering type (average measures) and definition of relationship (consistency). ICC > 0.70 indicated moderate to good reliability.

3) An **analysis of process indicators** described how stakeholders’ responses evolved through iterations and how consensus or certain level of agreement has formed.
Stability of response presented the consistency of responses within the study period and between respondent group stability, which is considered a necessary precondition for determining the level of agreement or if consensus was achieved.
We used the coefficient of quartile variation (CQV) as a descriptive measure of response stability. CQV was calculated over all participants (CQVtotal) and within the same stakeholder group (CQVsub) to account for expected higher variation in response between different stakeholder groups. A CQVtotal less than 30% and CQVsub less than 15% was considered as stable response.


```{r setup, include=FALSE}

# Importing packages needed for the analysis
library(mice)
library(tableone)
library(captioner)
library(kableExtra)
library(knitr)
library(tidyverse) # used for data cleaning
library(grid) # used to insert additional text on graphic
library(irr)
library(lmerTest)
library(performance)


# functions for add captioning for figures and tables in the markdown doc (as here: https://www.r-bloggers.com/r-markdown-how-to-number-and-reference-tables/)

table_nums <- captioner::captioner(prefix = "Table")
figure_nums <- captioner::captioner(prefix = "Figure")
t.ref <- function(x) {
  stringr::str_extract(table_nums(x), "[^:]*")
}
f.ref <- function(x) {
  stringr::str_extract(figure_nums(x), "[^:]*")
}

# importing the data tables for all the days of the survey

days.survey <- c("11.01", "11.02", "11.03", "11.04", "11.05", "11.06", "11.07", "11.08", "11.09", "11.10", 
                "11.11", "11.12", "11.13", "11.14", "11.15", "11.16", "11.17", "11.18", "11.19", "11.20", 
                "11.21", "11.22", "11.23", "11.24", "11.25", "11.26", "11.27", "11.28", "11.29", "11.30", 
                "12.01", "12.02", "12.03", "12.04", "12.05", "12.06", "12.07", "12.08", "12.09", "12.10", 
                 "12.11", "12.12", "12.13", "12.14", "12.15", "12.16", "12.17", "12.18", "12.19", "12.20", 
                "12.21", "12.22", "12.23", "12.24", "12.25", "12.26", "12.27", "12.28", "12.29", "12.30", 
                "12.31", "01.01", "01.02", "01.03", "01.04", "01.05", "01.06", "01.07", "01.08", "01.09", "01.10", 
                 "01.11", "01.12", "01.13", "01.14", "01.15", "01.16", "01.17" )

for( n in days.survey)  
{
data <- read.table(file=paste("enable-r-panel-enable-r",n,".csv", sep = ""),
                               header=TRUE, # first row contains variable names
                              sep = "," #, dec = ",", strip.white = TRUE, fileEncoding="UTF-8-BOM"
                   ) # ; separated columns
#add the date value in a new column
data$date <- n;
datatablename <- paste("data",n, sep = "");
assign(datatablename, data);

}

# make a big table with all the data
data <- rbind(data11.01, data11.02, data11.03, data11.04, data11.05, data11.06, data11.07, data11.08, data11.09, data11.10, 
                data11.11, data11.12, data11.13, data11.14, data11.15, data11.16, data11.17, data11.18, data11.19, data11.20, 
                data11.21, data11.22, data11.23, data11.24, data11.25, data11.26, data11.27, data11.28, data11.29, data11.30, 
                data12.01, data12.02, data12.03, data12.04, data12.05, data12.06, data12.07, data12.08, data12.09, data12.10, 
                data12.11, data12.12, data12.13, data12.14, data12.15, data12.16, data12.17, data12.18, data12.19, data12.20, 
                data12.21, data12.22, data12.23, data12.24, data12.25, data12.26, data12.27, data12.28, data12.29, data12.30, 
                data12.31, data01.01, data01.02, data01.03, data01.04, data01.05, data01.06, data01.07, data01.08, data01.09,
              data01.10, data01.11, data01.12, data01.13, data01.14, data01.15, data01.16, data01.17)

# rename the columns to work with them easier
namescolumns <- c ("Replier.Id" ,"TestRelevance" ,"TestClarity",
                   "TestRevisions", "TestMissing.attributes","Q1.Gender", "Q2.Age", "Q3.Country", "Q4.Education", "Q5.Expertise", "research.education",
                   "healthcare.practitioner", "policy.decision.maker", "patient.perspective", "Health.IT.specialist" , "DefinitionAgreement" , "DefinitionClarity", 
                   "Revision.of.the.definition" , "Definition.Other.suggestions", "CompletenessFramework" , "D1.1.Product.and.provider.information.Relevance", 
                   "D1.1.Product.and.provider.information.Clarity", "D1.1.Product.and.provider.information.Revisions",  "D1.1.Product.and.provider.information.Missing", 
                   "D2.1.Target.use.scenario.Relevance",  "D2.1.Target.use.scenario.Clarity", "D2.1.Target.use.scenario.Revisions", "D2.1.Target.use.scenario.Missing", 
                   "D2.2.Target.health.conditions.Relevance", "D2.2.Target.health.conditions.Clarity", 
                   "D2.2.Target.health.conditions.Revisions",  "D2.2.Target.health.conditions.Missing", 
                   "D2.3..Medication.regimen.Relevance", "D2.3..Medication.regimen.Clarity", 
                   "D2.3..Medication.regimen.Revisions", "D2.3..Medication.regimen.Missing" , 
                   "D2.4.1..Phase.of.medication.adherence.Relevance", "D2.4.1..Phase.of.medication.adherence.Clarity",  
                   "D2.4.1..Phase.of.medication.adherence.argument" , 
                   "D2.4.2.A.Monitoring.measurement.type.of.management.Relevance", "D2.4.2.A.Monitoring.measurement.type.of.management.Clarity" , 
                   "D2.4.2.A.Monitoring.measurement.type.of.management.Revisions" , "D2.4.2.A.Monitoring.measurement.type.of.management.Missing", 
                   "D2.4.2.B.Support.intervention.type.of.management.Relevance",  "D2.4.2.B.Support.intervention.type.of.management.Clarity", 
                   "D2.4.2.B.Support.intervention.type.of.management.Revisions", "D2.4.2.B.Support.intervention.type.of.management.Missing" , 
                   "D2.4.2.B.1.Intervention.modes.of.delivery.Relevance" , "D2.4.2.B.1.Intervention.modes.of.delivery.Clarity" , 
                   "D2.4.2.B.1.Intervention.modes.of.delivery.Revisions" , "D2.4.2.B.1.Intervention.modes.of.delivery.Missing" , 
                   "D2.4.2.B.2.Target.behaviour.determinants..Relevance" , "D2.4.2.B.2.Target.behaviour.determinants..Clarity", 
                   "D2.4.2.B.2.Target.behaviour.determinants..Revisions" ,  "D2.4.2.B.2.Target.behaviour.determinants..Missing", 
                   "D2.4.2.B.3.Behaviour.change.techniques.Relevance",  "D2.4.2.B.3.Behaviour.change.techniques.Clarity", 
                   "D2.4.2.B.3.Behaviour.change.techniques.Revisions",  "D2.4.2.B.3.Behaviour.change.techniques.Missing", 
                   "D2.4.2.B.4.Intervention.provider.Relevance",  "D2.4.2.B.4.Intervention.provider.Clarity",
                   "D2.4.2.B.4.Intervention.provider.Revisions" ,  "D2.4.2.B.4.Intervention.provider.Missing" , 
                   "D2.4.2.B.5.Intervention.setting..Relevance", "D2.4.2.B.5.Intervention.setting..Clarity", 
                   "D2.4.2.B.5.Intervention.setting..Revisions",  "D2.4.2.B.5.Intervention.setting..Missing" , 
                   "D3.1.1.A.ISO.certification.Relevance",  "D3.1.1.A.ISO.certification.Clarity", 
                   "D3.1.1.A.ISO.certification.Revisions",  "D3.1.1.A.ISO.certification.Missing" , 
                   "D3.1.1.B.Evidence.from.scientific.evaluation.Relevance",  "D3.1.1.B.Evidence.from.scientific.evaluation.Clarity" , 
                   "D3.1.1.B.Evidence.from.scientific.evaluation.Revisions" ,  "D3.1.1.B.Evidence.from.scientific.evaluation.Missing" , 
                   "D3.1.1.C.Development.standards.Relevance" ,  "D3.1.1.C.Development.standards.Clarity", 
                   "D3.1.1.C.Development.standards.Revisions" ,  "D3.1.1.C.Development.standards.Missing" , 
                   "D3.1.1.D.Technological.standards.Relevance" ,  "D3.1.1.D.Technological.standards.Clarity", 
                   "D3.1.1.D.Technological.standards.Revisions", "D3.1.1.D.Technological.standards.Missing", 
                   "D3.1.2.Research.related.quality.indicators.Relevance" ,  "D3.1.2.Research.related.quality.indicators.Clarity" , 
                   "D3.1.2.Research.related.quality.indicators.Revisions" ,  "D3.1.2.Research.related.quality.indicators.Missing", 
                   "D3.1.3.Policy.related.quality.indicators.Relevance" , "D3.1.3.Policy.related.quality.indicators.Clarity", 
                   "D3.1.3.Policy.related.quality.indicators.Revisions",  "D3.1.3.Policy.related.quality.indicators.Missing" , 
                   "D3.1.4.Use.related.quality.indicators.Relevance" ,  "D3.1.4.Use.related.quality.indicators.Clarity", 
                   "D3.1.4.Use.related.quality.indicators.Revisions" , "D3.1.4.Use.related.quality.indicators.Missing" , 
                   "D3.2.1.Implementation.outcomes.Relevance" ,  "D3.2.1.Implementation.outcomes.Clarity" , 
                   "D3.2.1.Implementation.outcomes.Revisions" ,  "D3.2.1.Implementation.outcomes.Missing" , 
                   "D3.2.2.Implementation.strategies.Relevance" ,  "D3.2.2.Implementation.strategies.Clarity" , 
                   "D3.2.2.Implementation.strategies.Revisions" ,  "D3.2.2.Implementation.strategies.Missing", "date") 

names(data) <- namescolumns

# exclude all rows that don't even have gender and age recorded; from scrolling through the table - all who did not answer these questions did not continue
data <- data[data$Q1.Gender != "" &data$Q2.Age != "" , ]
table(data$Replier.Id, exclude=NULL)

# how many unique identifiers of participants? - people who accepted the invitation
length(unique(data$Replier.Id))

# give names to variables capturing sociodems and variables about ratings - for later use:

VarSD1 <-  c ("Q1.Gender", "Q2.Age", "Q3.Country", "Q4.Education", "Q5.Expertise", "research.education",
                   "healthcare.practitioner", "policy.decision.maker", "patient.perspective", "Health.IT.specialist") 

VarRatings <-  c ( "DefinitionAgreement" , "DefinitionClarity", 
                   "D1.1.Product.and.provider.information.Relevance", 
                   "D1.1.Product.and.provider.information.Clarity",  
                   "D2.1.Target.use.scenario.Relevance",  "D2.1.Target.use.scenario.Clarity", 
                   "D2.2.Target.health.conditions.Relevance", "D2.2.Target.health.conditions.Clarity", 
                   "D2.3..Medication.regimen.Relevance", "D2.3..Medication.regimen.Clarity", 
                   "D2.4.1..Phase.of.medication.adherence.Relevance", "D2.4.1..Phase.of.medication.adherence.Clarity",  
                   "D2.4.2.A.Monitoring.measurement.type.of.management.Relevance", "D2.4.2.A.Monitoring.measurement.type.of.management.Clarity" , 
                   "D2.4.2.B.Support.intervention.type.of.management.Relevance",  "D2.4.2.B.Support.intervention.type.of.management.Clarity",
                   "D2.4.2.B.1.Intervention.modes.of.delivery.Relevance" , "D2.4.2.B.1.Intervention.modes.of.delivery.Clarity" , 
                   "D2.4.2.B.2.Target.behaviour.determinants..Relevance" , "D2.4.2.B.2.Target.behaviour.determinants..Clarity", 
                   "D2.4.2.B.3.Behaviour.change.techniques.Relevance",  "D2.4.2.B.3.Behaviour.change.techniques.Clarity", 
                   "D2.4.2.B.4.Intervention.provider.Relevance",  "D2.4.2.B.4.Intervention.provider.Clarity",
                   "D2.4.2.B.5.Intervention.setting..Relevance", "D2.4.2.B.5.Intervention.setting..Clarity", 
                   "D3.1.1.A.ISO.certification.Relevance",  "D3.1.1.A.ISO.certification.Clarity", 
                   "D3.1.1.B.Evidence.from.scientific.evaluation.Relevance",  "D3.1.1.B.Evidence.from.scientific.evaluation.Clarity" , 
                   "D3.1.1.C.Development.standards.Relevance" ,  "D3.1.1.C.Development.standards.Clarity", 
                   "D3.1.1.D.Technological.standards.Relevance" ,  "D3.1.1.D.Technological.standards.Clarity", 
                   "D3.1.2.Research.related.quality.indicators.Relevance" ,  "D3.1.2.Research.related.quality.indicators.Clarity" , 
                   "D3.1.3.Policy.related.quality.indicators.Relevance" , "D3.1.3.Policy.related.quality.indicators.Clarity", 
                   "D3.1.4.Use.related.quality.indicators.Relevance" ,  "D3.1.4.Use.related.quality.indicators.Clarity", 
                   "D3.2.1.Implementation.outcomes.Relevance" ,  "D3.2.1.Implementation.outcomes.Clarity" , 
                   "D3.2.2.Implementation.strategies.Relevance" ,  "D3.2.2.Implementation.strategies.Clarity" ) 


# EDelphi export ratings as 0-8, even if the labels are 1-9. Therefore we need to add +1 to all rating items before making any other move
for (n in VarRatings)
     {
      data[,n] <-  data[,n] +1;
 #     cat( "\n", n, ":" );
  #    print(table( data[,n] , exclude=NULL))
     }

# how many participants have answers on the 17.01 (date of closing the survey) to the last questions on implem strategies? - people who finished the survey
lastdayfulldata <- data[data$date=="01.17" & !is.na(data$D3.2.2.Implementation.strategies.Clarity), ]
nrow(lastdayfulldata)

# to check, how many answers with gender and age in the last day data? - people who has started the survey by the last day
names(data01.17) <- namescolumns
lastdaysomedata <- data01.17[data01.17$Q1.Gender != "" & data01.17$Q2.Age != "" ,]
nrow(lastdaysomedata)

# between those who really started with sociodemographics and those who completed, we might be able to use more of those with incomplete data - check how many:

# check those with ratings given on the first question (definitions)
lastdaydefnodata <- lastdaysomedata[is.na(lastdaysomedata$DefinitionAgreement) ,]
nrow(lastdaydefnodata)

# 10 people did not give any answers - the rest have at least the definition rating --> keep all with at least 1 rating in a separate dataset
lastdaysomedata <- data01.17[data01.17$Q1.Gender != "" & data01.17$Q2.Age != "" & !is.na(data01.17$DefinitionAgreement), -c(2:5)]
nrow(lastdaysomedata)

# redo changes for 1-9 in this sample too:
for (n in VarRatings)
     {
      lastdaysomedata[,n] <-  lastdaysomedata[,n] +1;
      cat( "\n", n, ":" );
      print(table( lastdaysomedata[,n] , exclude=NULL))
     }

# check missing data for items with ratings & sociodems:

md.pattern(lastdaysomedata[,VarSD1], rotate.names = TRUE)
# cute cat face, no missings on sociodems

md.pattern(lastdaysomedata[,VarRatings], rotate.names = TRUE)
pMiss <- function(x){sum(is.na(x))/length(x)*100}
apply(lastdaysomedata[,VarRatings],2,pMiss)
table(apply(lastdaysomedata[,VarRatings],1,pMiss), exclude=NULL)
# some missings on ratings


#correct one value in Health.IT.specialist that somehow ended up with 2 values (?!?) - get bigger value
lastdaysomedata$Health.IT.specialist[lastdaysomedata$Health.IT.specialist =="less than 5 years experience,5 to <10 years experience"] <- "5 to <10 years experience"

#calculate number with res.ed hcp policy patient healthIT

lastdaysomedata$res.ed.nr <- 1
lastdaysomedata$res.ed.nr[lastdaysomedata$research.education ==""] <- 0

lastdaysomedata$hcp.nr <- 1
lastdaysomedata$hcp.nr[lastdaysomedata$healthcare.practitioner ==""] <- 0

lastdaysomedata$policy.nr <- 1
lastdaysomedata$policy.nr[lastdaysomedata$policy.decision.maker ==""] <- 0

lastdaysomedata$patient.nr <- 1
lastdaysomedata$patient.nr[lastdaysomedata$patient.perspective ==""] <- 0

lastdaysomedata$healthIT.nr <- 1
lastdaysomedata$healthIT.nr[lastdaysomedata$Health.IT.specialist ==""] <- 0

# 0 should be none with no expertise
lastdaysomedata$exp.nr <- rowSums(lastdaysomedata[, c("res.ed.nr", "hcp.nr", "policy.nr", "patient.nr" , "healthIT.nr")])
table(lastdaysomedata$exp.nr, exclude=NULL)
# none with no expertise, some identified with all expertise areas (this is a problem, because they were supposed to leave unanswered those where they don't have expertise)

# select those with more expertise areas & consider "" (0) for those ares in which they indicated 'less than 5 years expertise'
lastdaysomedata[lastdaysomedata$exp.nr == 5, VarSD1[6:10]]
# only 1 has >5 years in all areas, the rest have at least 1 area with <5 --> can consider they meant none? we leave as reported, assuming all understood the instructions

# # correct to 'none' below for those with <5
# lastdayokdata$research.education[lastdayokdata$exp.nr == 5 & lastdayokdata$research.education == "less than 5 years experience"] <- ""
# lastdayokdata$healthcare.practitioner[lastdayokdata$exp.nr == 5 & lastdayokdata$healthcare.practitioner == "less than 5 years experience"] <- ""
# lastdayokdata$policy.decision.maker[lastdayokdata$exp.nr == 5 & lastdayokdata$policy.decision.maker == "less than 5 years experience"] <- ""
# lastdayokdata$patient.perspective[lastdayokdata$exp.nr == 5 & lastdayokdata$patient.perspective == "less than 5 years experience"] <- ""
# lastdayokdata$Health.IT.specialist[lastdayokdata$exp.nr == 5 & lastdayokdata$Health.IT.specialist == "less than 5 years experience"] <- ""
# # recalculate dichotomous & total number
# lastdayokdata$res.ed.nr <- 1
# lastdayokdata$res.ed.nr[lastdayokdata$research.education ==""] <- 0
# lastdayokdata$hcp.nr <- 1
# lastdayokdata$hcp.nr[lastdayokdata$healthcare.practitioner ==""] <- 0
# lastdayokdata$policy.nr <- 1
# lastdayokdata$policy.nr[lastdayokdata$policy.decision.maker ==""] <- 0
# lastdayokdata$patient.nr <- 1
# lastdayokdata$patient.nr[lastdayokdata$patient.perspective ==""] <- 0
# lastdayokdata$healthIT.nr <- 1
# lastdayokdata$healthIT.nr[lastdayokdata$Health.IT.specialist ==""] <- 0
# lastdayokdata$exp.nr <- rowSums(lastdayokdata[, c("res.ed.nr", "hcp.nr", "policy.nr", "patient.nr" , "healthIT.nr")])
# # in this case there in 1 with no experience whatsoever, so it is not a good solution --> better leave all with what they have answered

# add the new vars to sociodemo
VarSD2 <- c(VarSD1, "res.ed.nr" , "hcp.nr", "policy.nr" , "patient.nr" , "healthIT.nr" , "exp.nr" )


# select those with less than 25% missing and sociodemographics, for sample description
percmiss <- apply(lastdaysomedata[,VarRatings],1,pMiss)
str(percmiss)
names(percmiss)[percmiss <= 25]
lastdayokdata <- lastdaysomedata[names(percmiss)[percmiss <= 25], ]
md.pattern(lastdayokdata[,VarRatings], rotate.names = TRUE)

# check descriptives last day full responses
for( n in VarSD1)  
{
  cat( "\n", n, ":" );
  print( table(lastdayokdata[,n], exclude=NULL ) );
}



```

# Participation rates, sample size and missing data analysis

The final sample includes `r nrow(data01.17)` participants from `r length(unique(data01.17$Q3.Country))` countries who connected at least once, of which `r nrow(lastdaysomedata)` participants started the survey. Of these, `r nrow(lastdayokdata)` participants from `r length(unique(lastdayokdata$Q3.Country))` countries have completed sociodemographic data and at least 75 percent of the rating questions. The descriptive statistics are presented for the latter group in `r t.ref("tab_1")`, and for all participants who started the survey in `r t.ref("tab_2")`. 


```{r descriptives OK, eval=TRUE, message=FALSE, warning=FALSE, include=FALSE, results='hide'}
#table one
myVars <- VarSD2
## Vector of categorical variables that need transformation
catVars <- VarSD2
## Create a TableOne object
tab1 <- CreateTableOne(vars = myVars, # all variables to include in table
                       data = lastdayokdata, # the dataset
                       factorVars = catVars, 
                       includeNA = TRUE
                      ) 
tabPrep <- print(tab1, 
      showAllLevels = TRUE, # if all factor levels should be included in the table for categorical variables
      # SMD=TRUE # to show standardized mean differences
      test=FALSE)
tabPrep  <- cbind("Values" = rownames(tabPrep), tabPrep)
rownames(tabPrep) <- NULL
colnames(tabPrep) <- c("", "Values", "Frequencies (%)")
tab.1_cap <- table_nums(name="tab_1", caption = "Descriptive statistics of sample characteristics")

```


```{r echo=FALSE, message=FALSE, warning=FALSE, results='asis'}
#show the descriptives table
 kable_styling(knitr::kable(tabPrep, caption = tab.1_cap),
 bootstrap_options = c("striped", "hover", "condensed", "responsive"))

```


```{r descriptives some, eval=TRUE, message=FALSE, warning=FALSE, include=FALSE, results='hide'}
#table one
myVars <- VarSD2
## Vector of categorical variables that need transformation
catVars <- VarSD2
## Create a TableOne object
tab1 <- CreateTableOne(vars = myVars, # all variables to include in table
                       data = lastdaysomedata, # the dataset
                       factorVars = catVars, 
                       includeNA = TRUE
                      ) 
tabPrep <- print(tab1, 
      showAllLevels = TRUE, # if all factor levels should be included in the table for categorical variables
      # SMD=TRUE # to show standardized mean differences
      test=FALSE)
tabPrep  <- cbind("Values" = rownames(tabPrep), tabPrep)
rownames(tabPrep) <- NULL
colnames(tabPrep) <- c("", "Values", "Frequencies (%)")
tab.2_cap <- table_nums(name="tab_2", caption = "Descriptive statistics of sample characteristics")

```


```{r echo=FALSE, message=FALSE, warning=FALSE, results='asis'}
#show the descriptives table
 kable_styling(knitr::kable(tabPrep, caption = tab.2_cap),
 bootstrap_options = c("striped", "hover", "condensed", "responsive"))

```



<!-- ```{r descriptives ratings with all OK, eval=TRUE, message=FALSE, warning=FALSE, include=FALSE, results='hide'} -->
<!-- #table one -->
<!-- myVars <- VarRatings -->
<!-- ## Vector of categorical variables that need transformation -->

<!-- ## Create a TableOne object -->
<!-- tab1 <- CreateTableOne(vars = myVars, # all variables to include in table -->
<!--                        data = lastdayokdata, # the dataset -->
<!--                        factorVars = NULL,  -->
<!--                        includeNA = TRUE -->
<!--                       )  -->
<!-- tabPrep <- print(tab1, nonnormal = myVars, -->
<!--       showAllLevels = TRUE, # if all factor levels should be included in the table for categorical variables -->
<!--       # SMD=TRUE # to show standardized mean differences -->
<!--       test=FALSE) -->
<!-- tabPrep  <- cbind("Values" = rownames(tabPrep), tabPrep) -->
<!-- rownames(tabPrep) <- NULL -->
<!-- colnames(tabPrep) <- c("", "Values", "Median (IQR)") -->
<!-- tab.3_cap <- table_nums(name="tab_3", caption = "Descriptive statistics rating variables") -->

<!-- ``` -->



<!-- ```{r echo=FALSE, message=FALSE, warning=FALSE, results='asis'} -->
<!-- #show the descriptives table -->
<!--  kable_styling(knitr::kable(tabPrep, caption = tab.3_cap), -->
<!--  bootstrap_options = c("striped", "hover", "condensed", "responsive")) -->

<!-- ``` -->

# Descriptive graphs ratings

Here is an example of scatterplot for the MATech definition ratings. Should look just like the PDF report of the last day extraction.

```{r plots, fig.width=5, fig.height=5, warning=FALSE, message=FALSE, echo=FALSE}

ggplot(lastdaysomedata, aes(DefinitionAgreement, DefinitionClarity)) +
  ggtitle ("What is a Medication Adherence Technology?") +
  geom_point(size=3, color="blue", alpha=0.7) + 
  theme_bw() +
  theme(plot.title = element_text(color="black", size=12, face="bold.italic", hjust=0.5)) +
  scale_x_continuous(limits=c(1,9), breaks = seq(0, 9, by = 1)) +
  scale_y_continuous(limits=c(1,9), breaks = seq(0, 9, by =1 )) +
  xlab("Level of Agreement") + ylab("Clarity")

```

# Analysis ExpertLens


Descriptives rating variables, including participants who answered all questions are presented in `r f.ref("fig_1")`. 

```{r, echo=FALSE, include=FALSE, message=FALSE}

fig.1_cap <- figure_nums(name="fig_1", caption = "Panel ratings descriptor groups" )

fig.2_cap <- figure_nums(name="fig_2", caption = "Panel ratings descriptor groups HCP" )
fig.3_cap <- figure_nums(name="fig_3", caption = "Panel ratings descriptor groups decision makers" )
fig.4_cap <- figure_nums(name="fig_4", caption = "Panel ratings descriptor groups patient representatives" )
fig.5_cap <- figure_nums(name="fig_5", caption = "Panel ratings descriptor groups Health IT specialists" )
fig.6_cap <- figure_nums(name="fig_6", caption = "Panel ratings descriptor groups Research and Education" )


# ######################################################
#   adapted from AnalyzeExpertLens_Test.R
#   Pane (adapted from Smart) January 04 2021
#   Creates OPTIC Delphi Visualization Examples
#   data input: Excel file with data (cleaned)
#   data output: graphics
######################################################

# prepare definition data - from all who answered the definition
nrow(lastdaysomedata)
sum(is.na(lastdaysomedata$DefinitionAgreement))

# adapt to have longdata on Delphi questions with Outcome variable relevance = 1, clarity = 2 and Question as 1 - 20
lastdaysomedatalong <- gather(lastdaysomedata[,c("Q1.Gender","Q2.Age","Q3.Country","Q4.Education" ,"Q5.Expertise", "DefinitionAgreement", "DefinitionClarity")], 
                              item, rating, c("DefinitionAgreement", "DefinitionClarity"), factor_key=TRUE)
# label outcomes
lastdaysomedatalong_lab <- lastdaysomedatalong %>%
  mutate(
    Outcome = case_when(
      str_detect(item, "Agreement") ~ "Agreement",
      str_detect(item, "Clarity") ~ "Clarity",
                                 TRUE ~ NA_character_) %>% 
               factor(., levels = c("Agreement", "Clarity")))

DefRatings <- c("Definition")
#Label Question Variables
  lastdaysomedatalong_lab = lastdaysomedatalong_lab %>%
    mutate(Question = case_when(str_detect(item,  "Definition")  ~ "Definition",
                              TRUE ~ NA_character_) %>%
             # If you want to change the order of the labels on the graphics,
             # then you should just reorder the following:
             factor(., levels = DefRatings))

  # Sort Data
  lastdaysomedatalong_lab = lastdaysomedatalong_lab %>%
    arrange(Question, Outcome)

# prepare descriptor groups data

# adapt to have longdata on Delphi questions with Outcome variable relevance = 1, clarity = 2 and Question as 1 - 20
lastdayokdatalong <- gather(lastdayokdata[,c(VarSD2, VarRatings)], item, rating, VarRatings[-c(1,2)], factor_key=TRUE)
# label outcomes
lastdayokdatalong_lab <- lastdayokdatalong %>%
  mutate(
    Outcome = case_when(
      str_detect(item, "Relevance") ~ "Relevance",
      str_detect(item, "Clarity") ~ "Clarity",
                                 TRUE ~ NA_character_) %>% 
               factor(., levels = c("Relevance", "Clarity")))

revRatings <- rev(c("Product and provider information", "Target use scenario", "Target health conditions",
                             "Medication regimen","Medication adherence phase", "Medication adherence measurement",
                              "Medication adherence intervention", "Intervention mode of delivery", "Target behaviour determinants", "Behaviour change techniques",
                               "Intervention provider", "Intervention setting", "ISO certification",
                               "Scientific evaluation", "Development standards", "Technological standards",
                              "Research-related quality", "Policy-related quality", "Use-related quality",
                              "Implementation outcomes", "Implementation strategies"))
#Label Question Variables
  lastdayokdatalong_lab = lastdayokdatalong_lab %>%
    mutate(Question = case_when(str_detect(item,  "D1.1")  ~ "Product and provider information",
                              str_detect(item,  "D2.1") ~ "Target use scenario",
                              str_detect(item,  "D2.2") ~ "Target health conditions",
                              str_detect(item,  "D2.3") ~ "Medication regimen",
                              str_detect(item,  "D2.4.1") ~ "Medication adherence phase",
                              str_detect(item,  "D2.4.2.A") ~ "Medication adherence measurement",
                              str_detect(item,  "D2.4.2.B.S") ~ "Medication adherence intervention",
                              str_detect(item,  "D2.4.2.B.1") ~ "Intervention mode of delivery",
                              str_detect(item,  "D2.4.2.B.2") ~ "Target behaviour determinants",
                              str_detect(item,  "D2.4.2.B.3")  ~ "Behaviour change techniques",
                              str_detect(item,  "D2.4.2.B.4")  ~ "Intervention provider",
                              str_detect(item,  "D2.4.2.B.5")  ~ "Intervention setting",
                              str_detect(item,  "D3.1.1.A")  ~ "ISO certification",
                              str_detect(item,  "D3.1.1.B")  ~ "Scientific evaluation",
                              str_detect(item,  "D3.1.1.C")  ~ "Development standards",
                              str_detect(item,  "D3.1.1.D")  ~ "Technological standards",
                              str_detect(item,  "D3.1.2")  ~ "Research-related quality",
                              str_detect(item,  "D3.1.3")  ~ "Policy-related quality",
                              str_detect(item,  "D3.1.4")  ~ "Use-related quality",
                              str_detect(item,  "D3.2.1.")  ~ "Implementation outcomes",
                              str_detect(item,  "D3.2.2.")  ~ "Implementation strategies",
                              TRUE ~ NA_character_) %>%
             # If you want to change the order of the labels on the graphics,
             # then you should just reorder the following:
             factor(., levels = revRatings))

  # Sort Data
  lastdayokdatalong_lab = lastdayokdatalong_lab %>%
    arrange(Question, Outcome)

# function for summary
get_summary = function(data, question, outcome){
  ####################################
  # Load Function to get summary statistics
  # Inputs:
  # data - is the data.frame
  # question - is the column name in data representing the relevant question
  # outcome - is the column name in data representing the relevant outcome.
  ####################################
  
  # convert variables so can use in dplyr
  question = dplyr::enquo(question)
  outcome = dplyr::enquo(outcome)

  summary = data %>%
    # group data
    dplyr::group_by(!!question, !!outcome) %>%
    # get summaries by group (see group above)
    dplyr::summarize(median = median(rating, na.rm=T),
              LowerIPR = quantile(rating, probs=.3, na.rm=T),
              UpperIPR = quantile(rating, probs=.7, na.rm=T)) %>%
    # add columns to data.frame
    dplyr::mutate(IPRCP = (LowerIPR + UpperIPR)/2,
           AssymIndex = abs(5 - IPRCP),
           IPRAS = 2.35 + (1.5 * AssymIndex),
           IPR = UpperIPR - LowerIPR,
           DisagIndex = IPR / IPRAS,
           Disagreement = DisagIndex > 1)
  
  return(summary)
}

# function for graphic panel
graphic_panel = function(summary){
  ####################################
  # Load function to get Panel A graphic
  # Inputs:
  # summary - is the output from get_summary()
  ####################################
  
  # add variable for legend
summary <- cbind(summary, "color"= "1")
      scale_y = scale_y_continuous(
        name = "", #"Low           Moderate     High          Low   Moderate   High"
        breaks=c(1, 3, 7, 9),
        limits = c(1,9))  
      my_plot = ggplot(summary, aes(x = Question, y = median, colour = color
                                    )) +
        geom_point(position = position_dodge(.95)) + coord_flip() +
        facet_wrap(~ Outcome, nrow=1) +
        theme_bw() + 
        theme(axis.text.y = element_text( size = 10, face = "bold", colour = "#1A476F" ),
              strip.text = element_text(size=10, face = "bold"), 
              strip.background = element_rect(colour="black",
                                        fill="slategray2")) +
        scale_y + xlab("") +
        geom_hline(yintercept = 5) +
        geom_errorbar(aes(ymin=LowerIPR, ymax=UpperIPR), width=.2, 
                      position = position_dodge(.95)) + #position_dodge(.9)
   theme(legend.position = "bottom") +
          scale_color_manual(name = "", 
                             labels = "Rating: Median (30th-70th percentile)",
                             values = "#1A476F") +
          geom_vline(xintercept=9.5, colour="#808080", linetype="dashed") +
          geom_vline(xintercept=20.5, colour="#808080", linetype="dashed")

  return(my_plot)
}

# Load function to save graphic to pre-specified directory
save_graphic = function(file, graphic){
  ####################################
  # Load function to get Panel A graphic
  # Inputs:
  # file - is the location/name of your pdf graphic file. 
  # graphic - is the object created from graphic_panel()
  # panel - is the panel you want to produce a figure for. 
  # panel cont. - Possible options for panel are "A" or "B"
  # rating - The rating you want your figures for ("both", "R1_Rating", or "R3_Rating")
  ####################################
  png(file = file, width=8, height=8, units = "in", res = 300)

    print(graphic + geom_rect(aes(xmin = 0, xmax=22, ymin = 4 , ymax = 6), 
                              fill = "grey", color=alpha("grey",0), alpha=0.04), show.legend=F )
    grid.text("High", x = unit(0.61, "npc"), y = unit(0.109, "npc"), rot=0,
              gp = gpar(fontsize = 9, fontface = "plain"))
    grid.text("Moderate", x = unit(0.50, "npc"), y = unit(0.109, "npc"), rot=0,
              gp = gpar(fontsize = 9, fontface = "plain"))
    grid.text("Low", x = unit(0.39, "npc"), y = unit(0.109, "npc"), rot=0,
              gp = gpar(fontsize = 9, fontface = "plain"))
    
    grid.text("High", x = unit(0.94, "npc"), y = unit(0.109, "npc"), rot=0,
              gp = gpar(fontsize = 9, fontface = "plain"))
    grid.text("Moderate", x = unit(0.83, "npc"), y = unit(0.109, "npc"), rot=0,
              gp = gpar(fontsize = 9, fontface = "plain"))
    grid.text("Low", x = unit(0.72, "npc"), y = unit(0.109, "npc"), rot=0,
              gp = gpar(fontsize = 9, fontface = "plain"))
    
    # grid.text("High", x = unit(0.765, "npc"), y = unit(0.1, "npc"), rot=0,
    #           gp = gpar(fontsize = 9, fontface = "bold"))
    # grid.text("Moderate", x = unit(0.7, "npc"), y = unit(0.1, "npc"), rot=0,
    #           gp = gpar(fontsize = 9, fontface = "bold"))
    # grid.text("Low", x = unit(0.635, "npc"), y = unit(0.1, "npc"), rot=0,
    #           gp = gpar(fontsize = 9, fontface = "bold"))
    # 
    # grid.text("High", x = unit(0.96, "npc"), y = unit(0.1, "npc"), rot=0,
    #           gp = gpar(fontsize = 9, fontface = "bold"))
    # grid.text("Moderate", x = unit(0.895, "npc"), y = unit(0.1, "npc"), rot=0,
    #           gp = gpar(fontsize = 9, fontface = "bold"))
    # grid.text("Low", x = unit(0.83, "npc"), y = unit(0.1, "npc"), rot=0,
    #           gp = gpar(fontsize = 9, fontface = "bold"))

  dev.off()
}


# function for panel with several ratings (per group)



#### End of functions ####
```

```{r, echo=FALSE, include=FALSE, message=FALSE}
tab.3a_cap <- table_nums(name="tab_3a", caption = "Summary statistics definition ratings")

#### Panel plot ####
# Get Summary Statistics
summary_stats_Panel_def = get_summary(data = lastdaysomedatalong_lab, question = Question, outcome = Outcome)

```


Table with the agreement indices for the definition ratings:

```{r echo=FALSE, message=FALSE, warning=FALSE, results='asis'}
#show the descriptives table - here can exclude some columns by taking out their names from the c() in the first row below
 kable_styling(knitr::kable(summary_stats_Panel_def[, c("Question", "Outcome", "median", "LowerIPR", "UpperIPR", "IPRCP", "AssymIndex", "IPRAS", "IPR",  "DisagIndex", "Disagreement")], caption = tab.3a_cap),
 bootstrap_options = c("striped", "hover", "condensed", "responsive"))

```



```{r, echo=FALSE, include=FALSE, message=FALSE}
tab.3b_cap <- table_nums(name="tab_3b", caption = "Summary statistics descriptor group ratings")

#### Panel plot ####
# Get Summary Statistics
summary_stats_Panel = get_summary(data = lastdayokdatalong_lab, question = Question, outcome = Outcome)
# Create Panel B Graphic
plot_Panel1 = graphic_panel(summary = summary_stats_Panel) 
# Save Panel A Graphic
save_graphic(file = "PanelB_Round1_Ratings.png", graphic = plot_Panel1)

```

![`r fig.1_cap`](./PanelB_Round1_Ratings.png)
Table with the agreement indices:

```{r echo=FALSE, message=FALSE, warning=FALSE, results='asis'}
#show the descriptives table - here can exclude some columns by taking out their names from the c() in the first row below
 kable_styling(knitr::kable(summary_stats_Panel[, c("Question", "Outcome", "median", "LowerIPR", "UpperIPR", "IPRCP", "AssymIndex", "IPRAS", "IPR",  "DisagIndex", "Disagreement")], caption = tab.3b_cap),
 bootstrap_options = c("striped", "hover", "condensed", "responsive"))

```

```{r echo=FALSE, message=FALSE, warning=FALSE}


TableClarity <- round(summary_stats_Panel[summary_stats_Panel$Outcome =="Clarity", c("median", "LowerIPR", "UpperIPR", "IPR", "IPRAS", "DisagIndex")], 2)

TableClarity <- TableClarity[order(as.numeric(row.names(TableClarity)), decreasing=TRUE),]
TableClarity$Perc <- paste(TableClarity$LowerIPR, "-", TableClarity$UpperIPR)
TableClarity <- TableClarity[,c("median", "Perc", "IPR", "IPRAS", "DisagIndex")]
write.csv(TableClarity, 'tableclarity.csv')


TableRelevance <- round(summary_stats_Panel[summary_stats_Panel$Outcome =="Relevance", c("median", "LowerIPR", "UpperIPR", "IPR", "IPRAS", "DisagIndex")], 2)

TableRelevance <- TableRelevance[order(as.numeric(row.names(TableRelevance)), decreasing=TRUE),]
TableRelevance$Perc <- paste(TableRelevance$LowerIPR, "-", TableRelevance$UpperIPR)
TableRelevance <- TableRelevance[,c("median", "Perc", "IPR", "IPRAS", "DisagIndex")]
write.csv(TableRelevance, 'tablerelevance.csv')

```
## Subgroup analyses

### Descriptives stratified by stakeholder group. 

#### Healthcare practitioners

```{r echo=FALSE, message=FALSE, warning=FALSE}

# select HCP vs others
lastdaysomedataHCP <- lastdaysomedata[lastdaysomedata$healthcare.practitioner!="", ]
lastdaysomedataHCPother <- lastdaysomedata[lastdaysomedata$healthcare.practitioner=="", ]

# for attribute groups - only full data

lastdayokdataHCP <- lastdayokdata[lastdayokdata$healthcare.practitioner!="", ]
lastdayokdataHCPother <- lastdayokdata[lastdayokdata$healthcare.practitioner=="", ]


# redo plots only with HCPs 
# adapt to have longdata on Delphi questions with Outcome variable relevance = 1, clarity = 2 and Question as 1 - 20
lastdaysomedataHCPlong <- gather(lastdaysomedataHCP[,c("Q1.Gender","Q2.Age","Q3.Country","Q4.Education" ,"Q5.Expertise", "DefinitionAgreement", "DefinitionClarity")], 
                              item, rating, c("DefinitionAgreement", "DefinitionClarity"), factor_key=TRUE)
# label outcomes
lastdaysomedataHCPlong_lab <- lastdaysomedataHCPlong %>%
  mutate(
    Outcome = case_when(
      str_detect(item, "Agreement") ~ "Agreement",
      str_detect(item, "Clarity") ~ "Clarity",
                                 TRUE ~ NA_character_) %>% 
               factor(., levels = c("Agreement", "Clarity")))

DefRatings <- c("Definition")
#Label Question Variables
  lastdaysomedataHCPlong_lab = lastdaysomedataHCPlong_lab %>%
    mutate(Question = case_when(str_detect(item,  "Definition")  ~ "Definition",
                              TRUE ~ NA_character_) %>%
             # If you want to change the order of the labels on the graphics,
             # then you should just reorder the following:
             factor(., levels = DefRatings))

  # Sort Data
  lastdaysomedataHCPlong_lab = lastdaysomedataHCPlong_lab %>%
    arrange(Question, Outcome)
  
  # adapt to have longdata on Delphi questions with Outcome variable relevance = 1, clarity = 2 and Question as 1 - 20
lastdayokdataHCPlong <- gather(lastdayokdataHCP[,c(VarSD2, VarRatings)], item, rating, VarRatings[-c(1,2)], factor_key=TRUE)
# label outcomes
lastdayokdataHCPlong_lab <- lastdayokdataHCPlong %>%
  mutate(
    Outcome = case_when(
      str_detect(item, "Relevance") ~ "Relevance",
      str_detect(item, "Clarity") ~ "Clarity",
                                 TRUE ~ NA_character_) %>% 
               factor(., levels = c("Relevance", "Clarity")))

#Label Question Variables
  lastdayokdataHCPlong_lab = lastdayokdataHCPlong_lab %>%
    mutate(Question = case_when(str_detect(item,  "D1.1")  ~ "Product and provider information",
                              str_detect(item,  "D2.1") ~ "Target use scenario",
                              str_detect(item,  "D2.2") ~ "Target health conditions",
                              str_detect(item,  "D2.3") ~ "Medication regimen",
                              str_detect(item,  "D2.4.1") ~ "Medication adherence phase",
                              str_detect(item,  "D2.4.2.A") ~ "Medication adherence measurement",
                              str_detect(item,  "D2.4.2.B.S") ~ "Medication adherence intervention",
                              str_detect(item,  "D2.4.2.B.1") ~ "Intervention mode of delivery",
                              str_detect(item,  "D2.4.2.B.2") ~ "Target behaviour determinants",
                              str_detect(item,  "D2.4.2.B.3")  ~ "Behaviour change techniques",
                              str_detect(item,  "D2.4.2.B.4")  ~ "Intervention provider",
                              str_detect(item,  "D2.4.2.B.5")  ~ "Intervention setting",
                              str_detect(item,  "D3.1.1.A")  ~ "ISO certification",
                              str_detect(item,  "D3.1.1.B")  ~ "Scientific evaluation",
                              str_detect(item,  "D3.1.1.C")  ~ "Development standards",
                              str_detect(item,  "D3.1.1.D")  ~ "Technological standards",
                              str_detect(item,  "D3.1.2")  ~ "Research-related quality",
                              str_detect(item,  "D3.1.3")  ~ "Policy-related quality",
                              str_detect(item,  "D3.1.4")  ~ "Use-related quality",
                              str_detect(item,  "D3.2.1.")  ~ "Implementation outcomes",
                              str_detect(item,  "D3.2.2.")  ~ "Implementation strategies",
                              TRUE ~ NA_character_) %>%
             # If you want to change the order of the labels on the graphics,
             # then you should just reorder the following:
             factor(., levels = revRatings))

  # Sort Data
  lastdayokdataHCPlong_lab = lastdayokdataHCPlong_lab %>%
    arrange(Question, Outcome)


```


```{r, echo=FALSE, include=FALSE, message=FALSE}
tab.3c_cap <- table_nums(name="tab_3c", caption = "Summary statistics definition ratings HCP")

#### Panel plot ####
# Get Summary Statistics
summary_stats_PanelHCP_def = get_summary(data = lastdaysomedataHCPlong_lab, question = Question, outcome = Outcome)

```

Table with the agreement indices for the definition ratings:

```{r echo=FALSE, message=FALSE, warning=FALSE, results='asis'}
#show the descriptives table - here can exclude some columns by taking out their names from the c() in the first row below
 kable_styling(knitr::kable(summary_stats_PanelHCP_def[, c("Question", "Outcome", "median", "LowerIPR", "UpperIPR", "IPRCP", "AssymIndex", "IPRAS", "IPR",  "DisagIndex", "Disagreement")], caption = tab.3c_cap),
 bootstrap_options = c("striped", "hover", "condensed", "responsive"))

```


```{r, echo=FALSE, include=FALSE, message=FALSE}
tab.3d_cap <- table_nums(name="tab_3d", caption = "Summary statistics descriptor group ratings HCP")

#### Panel plot ####
# Get Summary Statistics
summary_stats_PanelHCP = get_summary(data = lastdayokdataHCPlong_lab, question = Question, outcome = Outcome)
# Create Panel B Graphic
plot_Panel1HCP = graphic_panel(summary = summary_stats_PanelHCP) 
# Save Panel A Graphic
save_graphic(file = "PanelB_Round1_HCPRatings.png", graphic = plot_Panel1HCP)

```

![`r fig.2_cap`](./PanelB_Round1_HCPRatings.png)
Table with the agreement indices:

```{r echo=FALSE, message=FALSE, warning=FALSE, results='asis'}
#show the descriptives table - here can exclude some columns by taking out their names from the c() in the first row below
 kable_styling(knitr::kable(summary_stats_PanelHCP[, c("Question", "Outcome", "median", "LowerIPR", "UpperIPR", "IPRCP", "AssymIndex", "IPRAS", "IPR",  "DisagIndex", "Disagreement")], caption = tab.3d_cap),
 bootstrap_options = c("striped", "hover", "condensed", "responsive"))

```


#### Policy and decision makers

```{r echo=FALSE, message=FALSE, warning=FALSE}

# select HCP vs others
lastdaysomedataPDM <- lastdaysomedata[lastdaysomedata$policy.decision.maker!="", ]
lastdaysomedataPDMother <- lastdaysomedata[lastdaysomedata$policy.decision.maker=="", ]

# for attribute groups - only full data

lastdayokdataPDM <- lastdayokdata[lastdayokdata$policy.decision.maker!="", ]
lastdayokdataPDMother <- lastdayokdata[lastdayokdata$policy.decision.maker=="", ]


# redo plots only with HCPs 
# adapt to have longdata on Delphi questions with Outcome variable relevance = 1, clarity = 2 and Question as 1 - 20
lastdaysomedataPDMlong <- gather(lastdaysomedataPDM[,c("Q1.Gender","Q2.Age","Q3.Country","Q4.Education" ,"Q5.Expertise", "DefinitionAgreement", "DefinitionClarity")], 
                              item, rating, c("DefinitionAgreement", "DefinitionClarity"), factor_key=TRUE)
# label outcomes
lastdaysomedataPDMlong_lab <- lastdaysomedataPDMlong %>%
  mutate(
    Outcome = case_when(
      str_detect(item, "Agreement") ~ "Agreement",
      str_detect(item, "Clarity") ~ "Clarity",
                                 TRUE ~ NA_character_) %>% 
               factor(., levels = c("Agreement", "Clarity")))

DefRatings <- c("Definition")
#Label Question Variables
  lastdaysomedataPDMlong_lab = lastdaysomedataPDMlong_lab %>%
    mutate(Question = case_when(str_detect(item,  "Definition")  ~ "Definition",
                              TRUE ~ NA_character_) %>%
             # If you want to change the order of the labels on the graphics,
             # then you should just reorder the following:
             factor(., levels = DefRatings))

  # Sort Data
  lastdaysomedataPDMlong_lab = lastdaysomedataPDMlong_lab %>%
    arrange(Question, Outcome)
  
  # adapt to have longdata on Delphi questions with Outcome variable relevance = 1, clarity = 2 and Question as 1 - 20
lastdayokdataPDMlong <- gather(lastdayokdataPDM[,c(VarSD2, VarRatings)], item, rating, VarRatings[-c(1,2)], factor_key=TRUE)
# label outcomes
lastdayokdataPDMlong_lab <- lastdayokdataPDMlong %>%
  mutate(
    Outcome = case_when(
      str_detect(item, "Relevance") ~ "Relevance",
      str_detect(item, "Clarity") ~ "Clarity",
                                 TRUE ~ NA_character_) %>% 
               factor(., levels = c("Relevance", "Clarity")))

#Label Question Variables
  lastdayokdataPDMlong_lab = lastdayokdataPDMlong_lab %>%
    mutate(Question = case_when(str_detect(item,  "D1.1")  ~ "Product and provider information",
                              str_detect(item,  "D2.1") ~ "Target use scenario",
                              str_detect(item,  "D2.2") ~ "Target health conditions",
                              str_detect(item,  "D2.3") ~ "Medication regimen",
                              str_detect(item,  "D2.4.1") ~ "Medication adherence phase",
                              str_detect(item,  "D2.4.2.A") ~ "Medication adherence measurement",
                              str_detect(item,  "D2.4.2.B.S") ~ "Medication adherence intervention",
                              str_detect(item,  "D2.4.2.B.1") ~ "Intervention mode of delivery",
                              str_detect(item,  "D2.4.2.B.2") ~ "Target behaviour determinants",
                              str_detect(item,  "D2.4.2.B.3")  ~ "Behaviour change techniques",
                              str_detect(item,  "D2.4.2.B.4")  ~ "Intervention provider",
                              str_detect(item,  "D2.4.2.B.5")  ~ "Intervention setting",
                              str_detect(item,  "D3.1.1.A")  ~ "ISO certification",
                              str_detect(item,  "D3.1.1.B")  ~ "Scientific evaluation",
                              str_detect(item,  "D3.1.1.C")  ~ "Development standards",
                              str_detect(item,  "D3.1.1.D")  ~ "Technological standards",
                              str_detect(item,  "D3.1.2")  ~ "Research-related quality",
                              str_detect(item,  "D3.1.3")  ~ "Policy-related quality",
                              str_detect(item,  "D3.1.4")  ~ "Use-related quality",
                              str_detect(item,  "D3.2.1.")  ~ "Implementation outcomes",
                              str_detect(item,  "D3.2.2.")  ~ "Implementation strategies",
                              TRUE ~ NA_character_) %>%
             # If you want to change the order of the labels on the graphics,
             # then you should just reorder the following:
             factor(., levels = revRatings))

  # Sort Data
  lastdayokdataPDMlong_lab = lastdayokdataPDMlong_lab %>%
    arrange(Question, Outcome)


```


```{r, echo=FALSE, include=FALSE, message=FALSE}
tab.3e_cap <- table_nums(name="tab_3e", caption = "Summary statistics definition ratings PDM")

#### Panel plot ####
# Get Summary Statistics
summary_stats_PanelPDM_def = get_summary(data = lastdaysomedataPDMlong_lab, question = Question, outcome = Outcome)

```

Table with the agreement indices for the definition ratings:

```{r echo=FALSE, message=FALSE, warning=FALSE, results='asis'}
#show the descriptives table - here can exclude some columns by taking out their names from the c() in the first row below
 kable_styling(knitr::kable(summary_stats_PanelPDM_def[, c("Question", "Outcome", "median", "LowerIPR", "UpperIPR", "IPRCP", "AssymIndex", "IPRAS", "IPR",  "DisagIndex", "Disagreement")], caption = tab.3e_cap),
 bootstrap_options = c("striped", "hover", "condensed", "responsive"))

```


```{r, echo=FALSE, include=FALSE, message=FALSE}
tab.3f_cap <- table_nums(name="tab_3f", caption = "Summary statistics descriptor group ratings PDM")

#### Panel plot ####
# Get Summary Statistics
summary_stats_PanelPDM = get_summary(data = lastdayokdataPDMlong_lab, question = Question, outcome = Outcome)
# Create Panel B Graphic
plot_Panel1PDM = graphic_panel(summary = summary_stats_PanelPDM) 
# Save Panel A Graphic
save_graphic(file = "PanelB_Round1_PDMRatings.png", graphic = plot_Panel1PDM)

```

![`r fig.3_cap`](./PanelB_Round1_PDMRatings.png)
Table with the agreement indices:

```{r echo=FALSE, message=FALSE, warning=FALSE, results='asis'}
#show the descriptives table - here can exclude some columns by taking out their names from the c() in the first row below
 kable_styling(knitr::kable(summary_stats_PanelPDM[, c("Question", "Outcome", "median", "LowerIPR", "UpperIPR", "IPRCP", "AssymIndex", "IPRAS", "IPR",  "DisagIndex", "Disagreement")], caption = tab.3f_cap),
 bootstrap_options = c("striped", "hover", "condensed", "responsive"))

```


#### Patient representatives

```{r echo=FALSE, message=FALSE, warning=FALSE}

# select HCP vs others
lastdaysomedataPRP <- lastdaysomedata[lastdaysomedata$patient.perspective!="", ]
lastdaysomedataPRPother <- lastdaysomedata[lastdaysomedata$patient.perspective=="", ]

# for attribute groups - only full data

lastdayokdataPRP <- lastdayokdata[lastdayokdata$patient.perspective!="", ]
lastdayokdataPRPother <- lastdayokdata[lastdayokdata$patient.perspective=="", ]


# redo plots only with PRPs 
# adapt to have longdata on Delphi questions with Outcome variable relevance = 1, clarity = 2 and Question as 1 - 20
lastdaysomedataPRPlong <- gather(lastdaysomedataPRP[,c("Q1.Gender","Q2.Age","Q3.Country","Q4.Education" ,"Q5.Expertise", "DefinitionAgreement", "DefinitionClarity")], 
                              item, rating, c("DefinitionAgreement", "DefinitionClarity"), factor_key=TRUE)
# label outcomes
lastdaysomedataPRPlong_lab <- lastdaysomedataPRPlong %>%
  mutate(
    Outcome = case_when(
      str_detect(item, "Agreement") ~ "Agreement",
      str_detect(item, "Clarity") ~ "Clarity",
                                 TRUE ~ NA_character_) %>% 
               factor(., levels = c("Agreement", "Clarity")))

DefRatings <- c("Definition")
#Label Question Variables
  lastdaysomedataPRPlong_lab = lastdaysomedataPRPlong_lab %>%
    mutate(Question = case_when(str_detect(item,  "Definition")  ~ "Definition",
                              TRUE ~ NA_character_) %>%
             # If you want to change the order of the labels on the graphics,
             # then you should just reorder the following:
             factor(., levels = DefRatings))

  # Sort Data
  lastdaysomedataPRPlong_lab = lastdaysomedataPRPlong_lab %>%
    arrange(Question, Outcome)
  
  # adapt to have longdata on Delphi questions with Outcome variable relevance = 1, clarity = 2 and Question as 1 - 20
lastdayokdataPRPlong <- gather(lastdayokdataPRP[,c(VarSD2, VarRatings)], item, rating, VarRatings[-c(1,2)], factor_key=TRUE)
# label outcomes
lastdayokdataPRPlong_lab <- lastdayokdataPRPlong %>%
  mutate(
    Outcome = case_when(
      str_detect(item, "Relevance") ~ "Relevance",
      str_detect(item, "Clarity") ~ "Clarity",
                                 TRUE ~ NA_character_) %>% 
               factor(., levels = c("Relevance", "Clarity")))

#Label Question Variables
  lastdayokdataPRPlong_lab = lastdayokdataPRPlong_lab %>%
    mutate(Question = case_when(str_detect(item,  "D1.1")  ~ "Product and provider information",
                              str_detect(item,  "D2.1") ~ "Target use scenario",
                              str_detect(item,  "D2.2") ~ "Target health conditions",
                              str_detect(item,  "D2.3") ~ "Medication regimen",
                              str_detect(item,  "D2.4.1") ~ "Medication adherence phase",
                              str_detect(item,  "D2.4.2.A") ~ "Medication adherence measurement",
                              str_detect(item,  "D2.4.2.B.S") ~ "Medication adherence intervention",
                              str_detect(item,  "D2.4.2.B.1") ~ "Intervention mode of delivery",
                              str_detect(item,  "D2.4.2.B.2") ~ "Target behaviour determinants",
                              str_detect(item,  "D2.4.2.B.3")  ~ "Behaviour change techniques",
                              str_detect(item,  "D2.4.2.B.4")  ~ "Intervention provider",
                              str_detect(item,  "D2.4.2.B.5")  ~ "Intervention setting",
                              str_detect(item,  "D3.1.1.A")  ~ "ISO certification",
                              str_detect(item,  "D3.1.1.B")  ~ "Scientific evaluation",
                              str_detect(item,  "D3.1.1.C")  ~ "Development standards",
                              str_detect(item,  "D3.1.1.D")  ~ "Technological standards",
                              str_detect(item,  "D3.1.2")  ~ "Research-related quality",
                              str_detect(item,  "D3.1.3")  ~ "Policy-related quality",
                              str_detect(item,  "D3.1.4")  ~ "Use-related quality",
                              str_detect(item,  "D3.2.1.")  ~ "Implementation outcomes",
                              str_detect(item,  "D3.2.2.")  ~ "Implementation strategies",
                              TRUE ~ NA_character_) %>%
             # If you want to change the order of the labels on the graphics,
             # then you should just reorder the following:
             factor(., levels = revRatings))

  # Sort Data
  lastdayokdataPRPlong_lab = lastdayokdataPRPlong_lab %>%
    arrange(Question, Outcome)


```


```{r, echo=FALSE, include=FALSE, message=FALSE}
tab.3g_cap <- table_nums(name="tab_3g", caption = "Summary statistics definition ratings PRP")

#### Panel plot ####
# Get Summary Statistics
summary_stats_PanelPRP_def = get_summary(data = lastdaysomedataPRPlong_lab, question = Question, outcome = Outcome)

```

Table with the agreement indices for the definition ratings:

```{r echo=FALSE, message=FALSE, warning=FALSE, results='asis'}
#show the descriptives table - here can exclude some columns by taking out their names from the c() in the first row below
 kable_styling(knitr::kable(summary_stats_PanelPRP_def[, c("Question", "Outcome", "median", "LowerIPR", "UpperIPR", "IPRCP", "AssymIndex", "IPRAS", "IPR",  "DisagIndex", "Disagreement")], caption = tab.3g_cap),
 bootstrap_options = c("striped", "hover", "condensed", "responsive"))

```


```{r, echo=FALSE, include=FALSE, message=FALSE}
tab.3h_cap <- table_nums(name="tab_3h", caption = "Summary statistics descriptor group ratings PRP")

#### Panel plot ####
# Get Summary Statistics
summary_stats_PanelPRP = get_summary(data = lastdayokdataPRPlong_lab, question = Question, outcome = Outcome)
# Create Panel B Graphic
plot_Panel1PRP = graphic_panel(summary = summary_stats_PanelPRP) 
# Save Panel A Graphic
save_graphic(file = "PanelB_Round1_PRPRatings.png", graphic = plot_Panel1PRP)

```

![`r fig.4_cap`](./PanelB_Round1_PRPRatings.png)
Table with the agreement indices:

```{r echo=FALSE, message=FALSE, warning=FALSE, results='asis'}
#show the descriptives table - here can exclude some columns by taking out their names from the c() in the first row below
 kable_styling(knitr::kable(summary_stats_PanelPRP[, c("Question", "Outcome", "median", "LowerIPR", "UpperIPR", "IPRCP", "AssymIndex", "IPRAS", "IPR",  "DisagIndex", "Disagreement")], caption = tab.3h_cap),
 bootstrap_options = c("striped", "hover", "condensed", "responsive"))

```


#### Health IT specialists

```{r echo=FALSE, message=FALSE, warning=FALSE}

# select HCP vs others
lastdaysomedataHIT <- lastdaysomedata[lastdaysomedata$Health.IT.specialist!="", ]
lastdaysomedataHITother <- lastdaysomedata[lastdaysomedata$Health.IT.specialist=="", ]

# for attribute groups - only full data

lastdayokdataHIT <- lastdayokdata[lastdayokdata$Health.IT.specialist!="", ]
lastdayokdataHITother <- lastdayokdata[lastdayokdata$Health.IT.specialist=="", ]


# redo plots only with HITs 
# adapt to have longdata on Delphi questions with Outcome variable relevance = 1, clarity = 2 and Question as 1 - 20
lastdaysomedataHITlong <- gather(lastdaysomedataHIT[,c("Q1.Gender","Q2.Age","Q3.Country","Q4.Education" ,"Q5.Expertise", "DefinitionAgreement", "DefinitionClarity")], 
                              item, rating, c("DefinitionAgreement", "DefinitionClarity"), factor_key=TRUE)
# label outcomes
lastdaysomedataHITlong_lab <- lastdaysomedataHITlong %>%
  mutate(
    Outcome = case_when(
      str_detect(item, "Agreement") ~ "Agreement",
      str_detect(item, "Clarity") ~ "Clarity",
                                 TRUE ~ NA_character_) %>% 
               factor(., levels = c("Agreement", "Clarity")))

DefRatings <- c("Definition")
#Label Question Variables
  lastdaysomedataHITlong_lab = lastdaysomedataHITlong_lab %>%
    mutate(Question = case_when(str_detect(item,  "Definition")  ~ "Definition",
                              TRUE ~ NA_character_) %>%
             # If you want to change the order of the labels on the graphics,
             # then you should just reorder the following:
             factor(., levels = DefRatings))

  # Sort Data
  lastdaysomedataHITlong_lab = lastdaysomedataHITlong_lab %>%
    arrange(Question, Outcome)
  
  # adapt to have longdata on Delphi questions with Outcome variable relevance = 1, clarity = 2 and Question as 1 - 20
lastdayokdataHITlong <- gather(lastdayokdataHIT[,c(VarSD2, VarRatings)], item, rating, VarRatings[-c(1,2)], factor_key=TRUE)
# label outcomes
lastdayokdataHITlong_lab <- lastdayokdataHITlong %>%
  mutate(
    Outcome = case_when(
      str_detect(item, "Relevance") ~ "Relevance",
      str_detect(item, "Clarity") ~ "Clarity",
                                 TRUE ~ NA_character_) %>% 
               factor(., levels = c("Relevance", "Clarity")))

#Label Question Variables
  lastdayokdataHITlong_lab = lastdayokdataHITlong_lab %>%
    mutate(Question = case_when(str_detect(item,  "D1.1")  ~ "Product and provider information",
                              str_detect(item,  "D2.1") ~ "Target use scenario",
                              str_detect(item,  "D2.2") ~ "Target health conditions",
                              str_detect(item,  "D2.3") ~ "Medication regimen",
                              str_detect(item,  "D2.4.1") ~ "Medication adherence phase",
                              str_detect(item,  "D2.4.2.A") ~ "Medication adherence measurement",
                              str_detect(item,  "D2.4.2.B.S") ~ "Medication adherence intervention",
                              str_detect(item,  "D2.4.2.B.1") ~ "Intervention mode of delivery",
                              str_detect(item,  "D2.4.2.B.2") ~ "Target behaviour determinants",
                              str_detect(item,  "D2.4.2.B.3")  ~ "Behaviour change techniques",
                              str_detect(item,  "D2.4.2.B.4")  ~ "Intervention provider",
                              str_detect(item,  "D2.4.2.B.5")  ~ "Intervention setting",
                              str_detect(item,  "D3.1.1.A")  ~ "ISO certification",
                              str_detect(item,  "D3.1.1.B")  ~ "Scientific evaluation",
                              str_detect(item,  "D3.1.1.C")  ~ "Development standards",
                              str_detect(item,  "D3.1.1.D")  ~ "Technological standards",
                              str_detect(item,  "D3.1.2")  ~ "Research-related quality",
                              str_detect(item,  "D3.1.3")  ~ "Policy-related quality",
                              str_detect(item,  "D3.1.4")  ~ "Use-related quality",
                              str_detect(item,  "D3.2.1.")  ~ "Implementation outcomes",
                              str_detect(item,  "D3.2.2.")  ~ "Implementation strategies",
                              TRUE ~ NA_character_) %>%
             # If you want to change the order of the labels on the graphics,
             # then you should just reorder the following:
             factor(., levels = revRatings))

  # Sort Data
  lastdayokdataHITlong_lab = lastdayokdataHITlong_lab %>%
    arrange(Question, Outcome)


```


```{r, echo=FALSE, include=FALSE, message=FALSE}
tab.3i_cap <- table_nums(name="tab_3i", caption = "Summary statistics definition ratings HIT")

#### Panel plot ####
# Get Summary Statistics
summary_stats_PanelHIT_def = get_summary(data = lastdaysomedataHITlong_lab, question = Question, outcome = Outcome)

```

Table with the agreement indices for the definition ratings:

```{r echo=FALSE, message=FALSE, warning=FALSE, results='asis'}
#show the descriptives table - here can exclude some columns by taking out their names from the c() in the first row below
 kable_styling(knitr::kable(summary_stats_PanelHIT_def[, c("Question", "Outcome", "median", "LowerIPR", "UpperIPR", "IPRCP", "AssymIndex", "IPRAS", "IPR",  "DisagIndex", "Disagreement")], caption = tab.3i_cap),
 bootstrap_options = c("striped", "hover", "condensed", "responsive"))

```


```{r, echo=FALSE, include=FALSE, message=FALSE}
tab.3j_cap <- table_nums(name="tab_3j", caption = "Summary statistics descriptor group ratings HIT")

#### Panel plot ####
# Get Summary Statistics
summary_stats_PanelHIT = get_summary(data = lastdayokdataHITlong_lab, question = Question, outcome = Outcome)
# Create Panel B Graphic
plot_Panel1HIT = graphic_panel(summary = summary_stats_PanelHIT) 
# Save Panel A Graphic
save_graphic(file = "PanelB_Round1_HITRatings.png", graphic = plot_Panel1HIT)

```

![`r fig.5_cap`](./PanelB_Round1_HITRatings.png)
Table with the agreement indices:

```{r echo=FALSE, message=FALSE, warning=FALSE, results='asis'}
#show the descriptives table - here can exclude some columns by taking out their names from the c() in the first row below
 kable_styling(knitr::kable(summary_stats_PanelHIT[, c("Question", "Outcome", "median", "LowerIPR", "UpperIPR", "IPRCP", "AssymIndex", "IPRAS", "IPR",  "DisagIndex", "Disagreement")], caption = tab.3j_cap),
 bootstrap_options = c("striped", "hover", "condensed", "responsive"))

```

#### Research and education

```{r echo=FALSE, message=FALSE, warning=FALSE}

# select REP vs others
lastdaysomedataREP <- lastdaysomedata[lastdaysomedata$research.education!="", ]
lastdaysomedataREPother <- lastdaysomedata[lastdaysomedata$research.education=="", ]

# for attribute groups - only full data

lastdayokdataREP <- lastdayokdata[lastdayokdata$research.education!="", ]
lastdayokdataREPother <- lastdayokdata[lastdayokdata$research.education=="", ]


# redo plots only with REPs 
# adapt to have longdata on Delphi questions with Outcome variable relevance = 1, clarity = 2 and Question as 1 - 20
lastdaysomedataREPlong <- gather(lastdaysomedataREP[,c("Q1.Gender","Q2.Age","Q3.Country","Q4.Education" ,"Q5.Expertise", "DefinitionAgreement", "DefinitionClarity")], 
                              item, rating, c("DefinitionAgreement", "DefinitionClarity"), factor_key=TRUE)
# label outcomes
lastdaysomedataREPlong_lab <- lastdaysomedataREPlong %>%
  mutate(
    Outcome = case_when(
      str_detect(item, "Agreement") ~ "Agreement",
      str_detect(item, "Clarity") ~ "Clarity",
                                 TRUE ~ NA_character_) %>% 
               factor(., levels = c("Agreement", "Clarity")))

DefRatings <- c("Definition")
#Label Question Variables
  lastdaysomedataREPlong_lab = lastdaysomedataREPlong_lab %>%
    mutate(Question = case_when(str_detect(item,  "Definition")  ~ "Definition",
                              TRUE ~ NA_character_) %>%
             # If you want to change the order of the labels on the graphics,
             # then you should just reorder the following:
             factor(., levels = DefRatings))

  # Sort Data
  lastdaysomedataREPlong_lab = lastdaysomedataREPlong_lab %>%
    arrange(Question, Outcome)
  
  # adapt to have longdata on Delphi questions with Outcome variable relevance = 1, clarity = 2 and Question as 1 - 20
lastdayokdataREPlong <- gather(lastdayokdataREP[,c(VarSD2, VarRatings)], item, rating, VarRatings[-c(1,2)], factor_key=TRUE)
# label outcomes
lastdayokdataREPlong_lab <- lastdayokdataREPlong %>%
  mutate(
    Outcome = case_when(
      str_detect(item, "Relevance") ~ "Relevance",
      str_detect(item, "Clarity") ~ "Clarity",
                                 TRUE ~ NA_character_) %>% 
               factor(., levels = c("Relevance", "Clarity")))

#Label Question Variables
  lastdayokdataREPlong_lab = lastdayokdataREPlong_lab %>%
    mutate(Question = case_when(str_detect(item,  "D1.1")  ~ "Product and provider information",
                              str_detect(item,  "D2.1") ~ "Target use scenario",
                              str_detect(item,  "D2.2") ~ "Target health conditions",
                              str_detect(item,  "D2.3") ~ "Medication regimen",
                              str_detect(item,  "D2.4.1") ~ "Medication adherence phase",
                              str_detect(item,  "D2.4.2.A") ~ "Medication adherence measurement",
                              str_detect(item,  "D2.4.2.B.S") ~ "Medication adherence intervention",
                              str_detect(item,  "D2.4.2.B.1") ~ "Intervention mode of delivery",
                              str_detect(item,  "D2.4.2.B.2") ~ "Target behaviour determinants",
                              str_detect(item,  "D2.4.2.B.3")  ~ "Behaviour change techniques",
                              str_detect(item,  "D2.4.2.B.4")  ~ "Intervention provider",
                              str_detect(item,  "D2.4.2.B.5")  ~ "Intervention setting",
                              str_detect(item,  "D3.1.1.A")  ~ "ISO certification",
                              str_detect(item,  "D3.1.1.B")  ~ "Scientific evaluation",
                              str_detect(item,  "D3.1.1.C")  ~ "Development standards",
                              str_detect(item,  "D3.1.1.D")  ~ "Technological standards",
                              str_detect(item,  "D3.1.2")  ~ "Research-related quality",
                              str_detect(item,  "D3.1.3")  ~ "Policy-related quality",
                              str_detect(item,  "D3.1.4")  ~ "Use-related quality",
                              str_detect(item,  "D3.2.1.")  ~ "Implementation outcomes",
                              str_detect(item,  "D3.2.2.")  ~ "Implementation strategies",
                              TRUE ~ NA_character_) %>%
             # If you want to change the order of the labels on the graphics,
             # then you should just reorder the following:
             factor(., levels = revRatings))

  # Sort Data
  lastdayokdataREPlong_lab = lastdayokdataREPlong_lab %>%
    arrange(Question, Outcome)


```


```{r, echo=FALSE, include=FALSE, message=FALSE}
tab.3k_cap <- table_nums(name="tab_3k", caption = "Summary statistics definition ratings REP")

#### Panel plot ####
# Get Summary Statistics
summary_stats_PanelREP_def = get_summary(data = lastdaysomedataREPlong_lab, question = Question, outcome = Outcome)

```

Table with the agreement indices for the definition ratings:

```{r echo=FALSE, message=FALSE, warning=FALSE, results='asis'}
#show the descriptives table - here can exclude some columns by taking out their names from the c() in the first row below
 kable_styling(knitr::kable(summary_stats_PanelREP_def[, c("Question", "Outcome", "median", "LowerIPR", "UpperIPR", "IPRCP", "AssymIndex", "IPRAS", "IPR",  "DisagIndex", "Disagreement")], caption = tab.3k_cap),
 bootstrap_options = c("striped", "hover", "condensed", "responsive"))

```


```{r, echo=FALSE, include=FALSE, message=FALSE}
tab.3l_cap <- table_nums(name="tab_3l", caption = "Summary statistics descriptor group ratings REP")

#### Panel plot ####
# Get Summary Statistics
summary_stats_PanelREP = get_summary(data = lastdayokdataREPlong_lab, question = Question, outcome = Outcome)
# Create Panel B Graphic
plot_Panel1REP = graphic_panel(summary = summary_stats_PanelREP) 
# Save Panel A Graphic
save_graphic(file = "PanelB_Round1_REPRatings.png", graphic = plot_Panel1REP)

```

![`r fig.6_cap`](./PanelB_Round1_REPRatings.png)
Table with the agreement indices:

```{r echo=FALSE, message=FALSE, warning=FALSE, results='asis'}
#show the descriptives table - here can exclude some columns by taking out their names from the c() in the first row below
 kable_styling(knitr::kable(summary_stats_PanelREP[, c("Question", "Outcome", "median", "LowerIPR", "UpperIPR", "IPRCP", "AssymIndex", "IPRAS", "IPR",  "DisagIndex", "Disagreement")], caption = tab.3l_cap),
 bootstrap_options = c("striped", "hover", "condensed", "responsive"))

```

### Reliability of ratings per group

The reliability of ratings per question across stakeholder groups was tested via intraclass correlation coefficient (ICC) based on two-way random model, considering type (average measures) and definition of relationship (consistency); median rating per subgroup taken as one rater.


```{r, echo=FALSE, include=FALSE, message=FALSE,  results='asis'}
tab.3m_cap <- table_nums(name="tab_3m", caption = "Median descriptor groups ratings for all subgroups")


summary_stats_PanelALL <- data.frame(summary_stats_PanelREP[, c("Question", "Outcome", "median")], summary_stats_PanelHCP$median, 
                                     summary_stats_PanelPDM$median,  summary_stats_PanelPRP$median,  summary_stats_PanelHIT$median)

names(summary_stats_PanelALL) <- c("Question", "Outcome", "REP.median", "HCP.median", "PDM.median", "PRP.median", "HIT.median")

```

```{r, echo=FALSE, message=FALSE,  results='asis'}

kable_styling(knitr::kable(summary_stats_PanelALL, caption = tab.3m_cap),
 bootstrap_options = c("striped", "hover", "condensed", "responsive"))

```


```{r, echo=FALSE, include=FALSE, message=FALSE}

ICCsubgroups <- irr::icc(summary_stats_PanelALL[,3:7], model = "twoway",
  type = "consistency", unit = "average")

ICCsubgroupsRel <- irr::icc(summary_stats_PanelALL[summary_stats_PanelALL$Outcome == "Relevance",3:7], model = "twoway",
  type = "consistency", unit = "average")

ICCsubgroupsClar <- irr::icc(summary_stats_PanelALL[summary_stats_PanelALL$Outcome == "Clarity",3:7], model = "twoway",
  type = "consistency", unit = "average")

```

Across items (relevance and clarity included), the ICC value across the median ratings of the `r ICCsubgroups$raters` subgroups was `r round(ICCsubgroups$value, 2)`, indicating good reliability. For Relevance, the ICC was `r round(ICCsubgroupsRel$value, 2)` and for Clarity it was `r round(ICCsubgroupsClar$value, 2)`. These results show that, if we consider items as 'subjects of rating' and median value of groups as individual raters, inter-rater reliability is above the pre-established threshold of good reliability, meaning that stakeholder groups rated the relevance and clarity of the descriptor groups in similar ways. 


```{r echo=FALSE, include=F,message=FALSE, warning=FALSE}

# patient.nr (18) < policy.nr (19) < healthIT.nr (24) < hcp.nr (45) < res.ed.nr (50)  (but last 2 groups overlap++)

lastdayokdata$group <- "hcp"
lastdayokdata$group[lastdayokdata$res.ed.nr == 1] <- "res.ed"
lastdayokdata$group[lastdayokdata$healthIT.nr == 1] <- "healthIT"
lastdayokdata$group[lastdayokdata$policy.nr == 1] <- "policy"
lastdayokdata$group[lastdayokdata$patient.nr == 1] <- "patient"

table(lastdayokdata$group, exclude=NULL)


lastdayokdatalong$group <- "hcp"
lastdayokdatalong$group[lastdayokdatalong$res.ed.nr == 1] <- "res.ed"
lastdayokdatalong$group[lastdayokdatalong$healthIT.nr == 1] <- "healthIT"
lastdayokdatalong$group[lastdayokdatalong$policy.nr == 1] <- "policy"
lastdayokdatalong$group[lastdayokdatalong$patient.nr == 1] <- "patient"

table(lastdayokdatalong$group, exclude=NULL)/42
lastdayokdatalong$group <- as.factor(lastdayokdatalong$group)

# for one item
model <- lmer(rating ~ (1 | group), data=lastdayokdatalong[lastdayokdatalong$item == names(table(lastdayokdatalong$item)[1]),])
performance::icc(model)

```

```{r, echo=FALSE, message=FALSE, warning=FALSE, results='asis'}

# for all items & definitions

# adapt to have longdata on Delphi questions with Outcome variable relevance = 1, clarity = 2 and Question as 1 - 20
lastdayokdatadeflong <- gather(lastdayokdata[,c(VarSD2, VarRatings)], item, rating, all_of(VarRatings), factor_key=TRUE)

# calculate groups for this dataset as well, for later


lastdayokdatadeflong$group <- "hcp"
lastdayokdatadeflong$group[lastdayokdatadeflong$res.ed.nr == 1] <- "res.ed"
lastdayokdatadeflong$group[lastdayokdatadeflong$healthIT.nr == 1] <- "healthIT"
lastdayokdatadeflong$group[lastdayokdatadeflong$policy.nr == 1] <- "policy"
lastdayokdatadeflong$group[lastdayokdatadeflong$patient.nr == 1] <- "patient"

table(lastdayokdatadeflong$group, exclude=NULL)/42
lastdayokdatadeflong$group <- as.factor(lastdayokdatadeflong$group)


for( n in 1: length(names(table(lastdayokdatalong$item))))  
{
model <- lmer(rating ~ (1 | group), data=lastdayokdatadeflong[lastdayokdatadeflong$item == names(table(lastdayokdatadeflong$item)[n]),], na.action = "na.exclude");
  cat( "\n", names(table(lastdayokdatadeflong$item)[n]), ":\n" );
  cat("Checking singularity:", performance::check_singularity(model), "\n" );
  print(performance::icc(model));
}

```

To examine whether there are any differences specific to each item, 2-level models were performed for each item, considering respondents nested within stakeholder group. Since 34 of 83 respondents indicated more than 1 group, it is necessary to allocate one single group for each of of the 34 respondents. One option is to maximize the equal distribution of participants in groups by allocating those with >2 groups to the one least represented. This decision resulted in `r sum(lastdayokdata$group == "res.ed")` respondents in the research and education group, `r sum(lastdayokdata$group == "hcp")` in the HCP group, `r sum(lastdayokdata$group == "healthIT")` in the healthIT group, `r sum(lastdayokdata$group == "policy")` in the policy group, and `r sum(lastdayokdata$group == "patient")` in the patient group.

The proportion of variance explained by the second level (the stakeholder group) was for most items either 0 (model almost/near singular) or below 0.1, with only two items with ICC above 0.15, i.e. 0.19 (target health conditions relevance) and 0.205 (intervention setting relevance). This indicates that >/= 80 percent of variance is within groups and therefore the differences across groups do not justify further investigation per item. 

# Temporal analysis of response rates and stability (analysis of process indicators)

This analysis examined how stakeholders’ responses evolved through iterations and if and how consensus or certain level of agreement has formed.

Stability of response refers to the consistency of responses within the study period and to between respondent group stability, and has been assessed by the coefficient of quartile variation (CQV). 

We have explored visually the evolution during the 78 survey days with two types of visualisations: for each question, the evolution of each respondent ratings color-coded by stakeholder group, and for each stakeholder group the evolution of median values of ratings. 

CQV was calculated with data from the last day of survey over all participants (CQVtotal) and within each stakeholder group (CQVsub) to account for expected higher variation in response between different stakeholder groups. 

A CQVtotal less than 30% and CQVsub less than 15% was considered as stable response.


```{r echo=FALSE, include=FALSE, message=FALSE, warning=FALSE}

# Visualisation of responses in different weeks of the delphi
# build the long dataset with all weeks, same as for the last week


# define the days as number of days since first survey result download

data$day.nr <- car::recode(data$date, 
                             '"11.01"=1;
                             "11.02"= 2;
                             "11.03"= 3;
                             "11.04"= 4;
                             "11.05"= 5;
                             "11.06"= 6;
                             "11.07"= 7;
                             "11.08"= 8;
                             "11.09"= 9;
                             "11.10"= 10;
                             "11.11"= 11;
                "11.12"= 12;
                "11.13"= 13;
                "11.14"= 14;
                "11.15"= 15;
                "11.16"= 16;
                "11.17"= 17;
                "11.18"= 18;
                "11.19"= 19;
                "11.20"= 20;                              
                "11.21"= 21;
                "11.22"= 22;
                "11.23"= 23;
                "11.24"= 24;
                "11.25"= 25;
                "11.26"= 26;
                "11.27"= 27;
                "11.28"= 28;
                "11.29"= 29;
                "11.30"= 30;                              
                "12.01"= 31;
                "12.02"= 32;
                "12.03"= 33;
                "12.04"= 34;
                "12.05"= 35;
                "12.06"= 36;
                "12.07"= 37;
                "12.08"= 38;
                "12.09"= 39;
                "12.10"= 40;
                 "12.11"= 41;
                 "12.12"= 42;
                 "12.13"= 43;
                 "12.14"= 44;
                 "12.15"= 45;
                 "12.16"= 46;
                 "12.17"= 47;
                 "12.18"= 48;
                 "12.19"= 49;
                 "12.20"= 50;  
                "12.21"= 51;
                "12.22"= 52; 
                "12.23"= 53;
                "12.24"= 54;
                "12.25"= 55;
                "12.26"= 56;
                "12.27"= 57; 
                "12.28"= 58; 
                "12.29"= 59; 
                "12.30"= 60; 
                "12.31"= 61; 
                "01.01"= 62;
                "01.02"= 63;
                "01.03"= 64; 
                "01.04"= 65; 
                "01.05"= 66; 
                "01.06"= 67; 
                "01.07"= 68;    
                "01.08"= 69;     
                "01.09"= 70;     
                "01.10"= 71;
                 "01.11"= 72; 
                "01.12"= 73;    
                "01.13"= 74; 
                "01.14"= 75; 
                "01.15"= 76;  
                "01.16"= 77;  
                "01.17"= 78')

unique(data$day.nr)


#  keep all with at least 1 rating in a separate dataset & exclude test items
somedata <- data[data$Q1.Gender != "" & data$Q2.Age != "" & !is.na(data$DefinitionAgreement), -c(2:5)]
nrow(somedata)

# check missing data for items with ratings & sociodems:

md.pattern(somedata[,VarSD1], rotate.names = TRUE)
# cute cat face, no missings on sociodems

md.pattern(somedata[,VarRatings], rotate.names = TRUE)
pMiss <- function(x){sum(is.na(x))/length(x)*100}
apply(somedata[,VarRatings],2,pMiss)
table(apply(somedata[,VarRatings],1,pMiss), exclude=NULL)
# some missings on ratings


# select those with less than 25% missing and sociodemographics, for sample description
percmiss <- apply(somedata[,VarRatings],1,pMiss)
str(percmiss)
names(percmiss)[percmiss <= 25]

okdata <- somedata[names(percmiss)[percmiss <= 25], ]
md.pattern(somedata[,VarRatings], rotate.names = TRUE)

nrow(okdata)

# make long format

# adapt to have longdata on Delphi questions with Outcome variable relevance = 1, clarity = 2 and Question as 1 - 20
okdatalong <- gather(okdata[,c(VarSD1, VarRatings, "Replier.Id", "day.nr")], item, rating, all_of(VarRatings), factor_key=TRUE)
okdatalonggr <- merge(okdatalong,lastdayokdata[,c("Replier.Id", "group") ],by="Replier.Id")

#separate first half from 2nd half of survey period
okdatalonggr$halfs <- 1
okdatalonggr$halfs[okdatalonggr$day.nr >= max(okdatalonggr$day.nr)/2 ] <- 2

```



```{r plotlong, fig.width=15, fig.height=30, warning=FALSE, message=FALSE, echo=FALSE}

# visualisation of all scores across weeks (spaghetti plots per question, lines colored by stakeholder group)
p <- ggplot(data = okdatalonggr, aes(x = day.nr, y = rating, group = Replier.Id))
p + theme_bw() +   scale_y_continuous(limits=c(1,9), breaks = seq(0, 9, by =1 )) + ggtitle ("Stability of ratings of individual respondents per question, color-coder per stakeholder group") + geom_line(aes(col = group)) + facet_wrap(. ~ item, nrow=11, ncol=4) 


```
The visualisation of individual ratings shows that most respondents did not change their ratings throughout the Delphi survey, and stakeholder groups had similar distribution of ratings across the 1 to 9 scale. 

```{r plotlong2, fig.width=10, fig.height=3, warning=FALSE, message=FALSE, echo=FALSE}

# summarize median and Q1 Q3 of ratings per day, item and group

summarybydate = okdatalonggr %>%
    # group data
    dplyr::group_by(day.nr, item, group) %>%
    # get summaries by group
    dplyr::summarize(median = median(rating, na.rm=T),
              Quar1 = quantile(rating, probs=.25, na.rm=T),
              Quar3 = quantile(rating, probs=.75, na.rm=T))


# New facet label names for group variable
group.labs <- c("HCP", "Technology developer", "Patient representative", "PDM", "Research")
names(group.labs) <- c("hcp", "healthIT", "patient", "policy", "res.ed")

# visualisation of all scores across weeks (spaghetti plots per question)
p <- ggplot(data = summarybydate, aes(x = day.nr, y = median, group = item))
p + theme_bw() +   scale_y_continuous(limits=c(1,9), breaks = seq(0, 9, by =1 )) + ggtitle ("Stability of median ratings per question, displayed separately for each stakeholder group") + geom_line() + facet_wrap(. ~ group, labeller = labeller(group = group.labs), nrow=1) + labs(x = "number of days since survey start")
```


```{r plotlong3, fig.width=10, fig.height=3, warning=FALSE, message=FALSE, echo=FALSE, include=FALSE}

# Save stability graphic:


  png(file = "StabilityRatings.png", width=5*4, height=4, units = "in", res = 300)
 p <- ggplot(data = summarybydate, aes(x = day.nr, y = median, group = item))
p + theme_bw() +   scale_y_continuous(limits=c(1,9), breaks = seq(0, 9, by =1 )) + geom_line() + facet_wrap(. ~ group, labeller = labeller(group = group.labs), nrow=1)+ labs(x = "number of days since survey start")
  dev.off()


```

The visualisation of the evolution across the survey period of the median responses for each item per stakeholder group shows that most items converge towards the last day on median ratings between 5 and 8 on a scale from 1 to 9. Therefore, across time we can see a stabilisation of the median values in the last half of the RT Delphi period. 

```{r echo=FALSE, include=FALSE, message=FALSE, warning=FALSE}

# summarize median Q1 Q3 of ratings across days, per item and group

summarybydate = okdatalonggr %>%
    # group data
    dplyr::group_by(day.nr, item, group) %>%
    # get summaries by group
    dplyr::summarize(median = median(rating, na.rm=T),
              Quar1 = quantile(rating, probs=.25, na.rm=T),
              Quar3 = quantile(rating, probs=.75, na.rm=T))


# calculate CQV per item within the same stakeholder group (CQVsub) for the last day

summarylastdayCQVsub = lastdayokdatadeflong %>%
    # group data
    dplyr::group_by(item, group) %>%
    # get summaries by group
    dplyr::summarize(median = median(rating, na.rm=T),
              Quar1 = quantile(rating, probs=.25, na.rm=T),
              Quar3 = quantile(rating, probs=.75, na.rm=T))

# CQV = (Q3-Q1)/(Q1+Q3)*100
summarylastdayCQVsub$CQVsub <- round(((summarylastdayCQVsub$Quar3 - summarylastdayCQVsub$Quar1)/(summarylastdayCQVsub$Quar3 + summarylastdayCQVsub$Quar1)*100), 2)

# calculate CQV per item over all participants (CQVtotal) for the last day

summarylastdayCQV = lastdayokdatadeflong %>%
    # group data
    dplyr::group_by(item) %>%
    # get summaries by group
    dplyr::summarize(median = median(rating, na.rm=T),
              Quar1 = quantile(rating, probs=.25, na.rm=T),
              Quar3 = quantile(rating, probs=.75, na.rm=T))

# CQV = (Q3-Q1)/(Q1+Q3)*100
summarylastdayCQV$CQVtotal <- round(((summarylastdayCQV$Quar3 - summarylastdayCQV$Quar1)/(summarylastdayCQV$Quar3 + summarylastdayCQV$Quar1)*100), 2)

# put group values in CQVtotal table (wide & merge)
summarylastdayCQVsubwide <-  spread(summarylastdayCQVsub, group, CQVsub)


summarylastdayCQVsubwide <- summarylastdayCQVsub[, c("item","group" , "CQVsub")] %>% 
  pivot_wider(names_from = group,values_from=CQVsub)

summarylastdayCQVall <- merge(summarylastdayCQV[, c("item", "CQVtotal")], summarylastdayCQVsubwide[,c("item"  ,"hcp"  ,    "healthIT" ,"patient" , "policy" ,  "res.ed") ],by="item")


tab.CQV_cap <- table_nums(name="tab_4", caption = "CQV values at closing of survey")

```


```{r echo=FALSE, message=FALSE, warning=FALSE, results='asis'}

#show the descriptives table
 kable_styling(knitr::kable(summarylastdayCQVall, caption = tab.CQV_cap),
 bootstrap_options = c("striped", "hover", "condensed", "responsive"))

```

All items showed stability across groups and the great majority also within groups, as per the thresholds established a priori. Some notable exceptions are the higher CQV values for Relevance on ISO Certification (all groups except patients), Intervention modes of delivery (healthIT), and target health conditions (patients). Clarity ratings for several attribute groups have been a matter of controversy within stakeholder groups. These results, together with the qualitative feedback received, would justify the improvement of the definitions and further development of extended explanations on the source of these terms and definitions and their practical applications.
