Elijah Robert Bouma-Sims
Lorrie Faith Cranor
Abstract
Websites implement cookie consent interfaces to obtain users’ permission to use non-essential cookies, as required by privacy regulations. We extend prior research evaluating the impact of interface design on cookie consent through an online behavioral experiment (n = 1359) in which we prompted mobile and desktop users from the UK and US to make cookie consent decisions using one of 14 interfaces implemented with the OneTrust consent management platform (CMP). We found significant effects on user behavior and sentiment for multiple explanatory variables, including more negative sentiment towards the consent process among UK participants and lower comprehension of interface information among mobile users. The design factor that had the largest effect on user behavior was the initial set of options displayed in the cookie banner. In addition to providing more evidence of the inadequacy of current cookie consent processes, our results have implications for website operators and CMPs.
1 INTRODUCTION
Cookie consent interfaces have become commonplace as websites attempt to comply with global privacy regulation. For example, the European Union’s 2002 ePrivacy directive (amended in 2009) requires that operators obtain opt-in consent prior to saving information to users’ devices, with a limited exception for “strictly necessary” cookies [6, 18]. The 2018 California Consumer Privacy Act (CCPA) requires opt-out consent for the collection of personal information [37]. Although privacy regulations generally require consent interfaces to be usable, no universal standards for the design of usable consent interfaces exist. Most websites now outsource their consent interfaces to consent management platforms (CMPs), many of which offer a diversity of interface designs, including some with deceptive design patterns [20, 23, 34]. Despite the now global scope of consent regulation and the central role of CMPs, little research has investigated the consent experiences of users in multiple localities [2, 7], on mobile devices [4, 46], and using a CMP in an experimental design. In this paper, we present a study examining the differences in attitudes and behavior towards cookie consent interfaces between the United Kingdom (UK) and United States (US) as well as between mobile phone users and users of other devices. We also investigate how attitudes and behavior vary based on key elements of CMP-implemented cookie banner design including banner prominence, location of cookie category definitions, and initial cookie options displayed. In particular, we seek to answer the following research questions:
(1) | Do users in the US and the UK interact with or perceive cookie consent interfaces differently? While users in the UK are subject to the UK GDPR, there is no comprehensive national privacy regulation in the US. We hypothesize that the different regulatory environments may cause users in each location to perceive cookie consent choices differently. | ||||
(2) | Do users on mobile devices interact with or perceive cookie consent interfaces differently as compared to desktop or laptop computer users? A large proportion of people access the internet using a mobile device or tablet; in June 2022, 52.36% of internet use in the US and 46.22% of internet use in the UK came from a mobile device [44, 45]. The restricted screen space, as compared to more traditional computing devices, often necessitates unique design considerations [8]. We hypothesize that mobile device users will have a less usable experience with cookie consent interfaces than those on other devices. | ||||
(3) | How do banner prominence, location of cookie category definitions, and initial cookie options impact attitudes and behavior towards cookie consent interfaces? A study conducted by Habib et al. [23] identified prominence and initial cookie options1 as commonly varied parameters in CMP-implemented cookie consent interfaces that effect usability. We hypothesized that adding the option to accept only strictly necessary cookies to the initial banner options, which often include only the options to accept all cookies or edit cookie preferences, may improve users’ ability to make their preferred decision without needing to review all available cookie options. We further hypothesized that displaying cookie category definitions on the initial cookie banner may improve user comprehension. | ||||
To explore these research questions, we extend the methods of Habib et al. [23] to conduct a user study comparing the cookie consent behavior and perceptions of crowdworkers located in the United States (US) and the United Kingdom (UK). We permitted participants to complete the study using mobile devices or computers. While completing a distraction task in a simulated e-commerce environment, participants were presented with one of 14 consent interface designs implemented using the OneTrust CMP, one of the most widely used CMPs [24]. These consent interface designs varied based on their prominence on the page, location of cookie category definitions, initial options, text layout, and approach to decision reversal. Once participants completed their task, they were directed to a post-task survey to evaluate cookie consent interface usability.
Using these methods, we observed a more negative perception of the consent process from UK-based participants and lower comprehension of consent interface information among participants on a mobile device. Of the design factors we evaluated, initial options had the largest effect on user experience; we failed to observe significant effects from notice prominence or location of definitions. We also observed confusion among participants about what would occur if they clicked the X to close the banner or otherwise did not make a decision. Ultimately, about half of our participants failed to make a consent decision that aligned with their reported preference. These results provide more evidence of the inadequacy of current consent processes and have implications for website operators and CMPs. Based on these results, we provide recommendations to improve the usability of cookie consent interfaces.
2 BACKGROUND AND RELATED WORK
In this section, we introduce background information and past research related to the impact of privacy laws on the cookie landscape, an overview of CMPs, and research on privacy choices on mobile devices and the impact of cookie consent interface design.
2.1 Legal Requirements
Cookie consent interfaces began to appear on websites as a response to a 2009 amendment to the EU’s ePrivacy Directive (EPD) that addressed confidentiality of digital communication and tracking on the internet [18]. Cookie consent interfaces became even more common with the European Union’s 2018 General Data Protection Regulation (GDPR) [17]. Under these rules, companies are required to obtain affirmative consent to process personal data beyond what is required to fulfill a legitimate business interest. This cannot be done through default settings or pre-checked boxes, and users must be able to withdraw their consent at any time. While the UK is no longer covered by the EU GDPR since the UK left the EU in 2020, the “key principles, rights and obligations remain the same” under the nearly identical UK GDPR [26]. Santos et al. identified requirements for cookie consent to comply with GDPR, including that the cookie consent interfaces should allow users to individually consent to cookies of different types, the consent must be informed, consent must be unambiguous, the banner should be reasonable (e. g. , easy to understand), and users should be able to revoke their consent at any time [40]. Research suggests that over 50% of cookie consent interfaces do not meet standards imposed by GDPR [1, 5, 31, 39, 41, 53].
The California Consumer Privacy Act (CCPA) went into effect in 2020, requiring (among other things) that certain companies provide notice to consumers related to data collection and offer them the ability to opt-out of the sale of personal information [37]. In practice, the CCPA’s notice and choice requirements are commonly provided through GDPR-style cookie consent banners [24]. The California Privacy Rights and Enforcement Act (CPRA), which took effect in January 2023, provides additional privacy rights to California consumers, including a right to opt-out of a business using sensitive personal information and to opt-out of the sharing of information with third parties [9].
2.2 Cookie Consent Interfaces and Consent Management Platforms
A variety of cookie consent interface designs are implemented on websites, including some that give users the options to opt-in to using cookies, some that allow users to decline cookies, and others that notify users that cookies are being used but do not allow any choice [13, 39, 53]. Information provided in cookie consent interfaces is not always accurate. For example, some websites misstate the reason for data collection or do not provide a reason at all [5, 41]. Researchers have found variations in location and styling, as well as the significant presence of nudging, pre-selection, and emphasized buttons to accept cookies [31, 32, 53].
Websites often turn to CMPs to generate cookie consent interfaces and record consent choices. While there are many CMPs [25], a small group are used by most websites. CMPs have become more prevalent over time with sharp upticks in their adoptions corresponding to new privacy laws going into effect. OneTrust has become the most used CMP since 2020 with options that can be tailored to comply with the CCPA [24]. These increases in adoption corresponding to when regulations go into effect suggest that legal compliance plays a large part in the adoption of CMPs. Even so, studies have found non-compliant consent interfaces implemented by CMPs on websites [13, 34, 49].
There is sometimes a difference between the cookie experience of users who are protected by privacy laws and those outside of their jurisdiction. For example, Dabrowski et al. found that 26.6% of sites that used cookies issued cookies for US-based visitors but not for EU-based ones after the GDPR went into effect. [12]. Further, Elik et al. found that the top-level domains of websites often explained most variations in cookie consent interfaces regardless of user location. A notable exception was seen with the use of the.com domain, in which users from the EU were significantly more likely to see a cookie banner than users in the US [54]. Finally, Alharbi et al. found that cookie consent interfaces from European websites were the most compliant with design best practices [1].
Many sites remained non-compliant with GDPR by failing to provide compliant cookie consent interfaces or incorrectly recording consent decisions [38]. Hils et al. examined 414 sites that used OneTrust as a CMP, observing that over 60% offered a non-compliant cookie consent interface that required a single click to accept cookies and a button or link that provided more granular options [24]. The privacy organization noyb has pushed for websites to adopt cookie consent interfaces that are compliant with the GDPR, encouraging the adoption of a three-button design that allows users to accept all cookies, reject all cookies, or access a secondary screen that allows for more granular control. [35]. French regulators have since stated that not having a button for rejecting cookies at the same level as one to accept them is not compliant with the ePrivacy Directive [28].
Cultural factors can influence how people perceive privacy risks [3, 11, 50]. In the context of cookies, Bornschein et al. found that the visibility of a cookie consent notice did not impact US participants’ risk perception while visibility did impact EU participants’ perception of risk on a website [7]. Bellentani found that users covered by GDPR chose to disclose more information after having seen a cookie consent interface relative to users who were not covered by GDPR [2]. Finally, Singh et al. found that whether the user is from the EU did not significantly influence their preferences for cookie consent interface design [42]. We further explore these differences by comparing the perception and consent behavior of participants located in the US and UK.
2.3 Mobile Devices
Compared to their full-sized counterparts, mobile devices have smaller screen sizes, use smaller fonts, have smaller buttons, and are used in more contexts [56]. In a study of AdChoices icon usability, Garlach et al. found that mobile devices further exacerbated problems with the icon. In particular, the decreased screen size made the icon more difficult to find and interact with [16]. Similarly, Singh et al. found that the smaller screen size made text-based privacy policies more difficult for users to understand on mobile platforms [43]. Users’ behavior concerning cookie consent interfaces also varies with the device type. Two studies found users on mobile devices were more likely to interact with consent interfaces, interact more quickly, accept the defaults provided to them, and accept cookies overall [4, 53]. On the other hand, Gunawan et al. found that notices on mobile platforms were more likely to have an option to reject cookies, which the authors postulate is due to the availability of APIs that can be used for other forms of tracking that do not involve cookies [21]. Our work builds on these studies by further exploring user perceptions and other usability aspects of the the consent experience, beyond consent decisions.
2.4 Impact of Cookie Consent Interface Design
Several studies have investigated how specific cookie consent interface design parameters, including deceptive patterns, impact the choices users make. Utz et al. found that users were more likely to interact with cookie consent interfaces on the lower left hand side of a screen [53] while Fernandez et al. found that location did not significantly impact users’ choice of setting [4]. Ma and Birrell found that banner text emphasizing the potential gains or losses of accepting cookies could influence users’ consent decisions [29]. However, when loss-aversion text was embedded in longer paragraphs, Habib et al. found no significant impact on consent decisions. Additionally, they found similar consent rates for interfaces with paragraph versus bullet-point text, and for banners with clearly-labeled buttons versus ambiguous buttons [23]. When users are shown confirmation-only interfaces or banners that only provide a binary choice they are more likely to consent to cookies than when they are presented with banners that allow opt-in to cookies by cookie type or vendor [53]. When specific cookie choices on the cookie consent banner were replaced with a “cookie preferences” button or link leading to a secondary interface with these choices, users were much more likely to accept all cookies [23]. Permissive default options lead to more types of cookies being accepted, with users being less sure of what options they had selected and less content with their choice when informed about what they chose [30].
3 METHODS
We conducted a between-subjects user study to compare participant behavior when interacting with one of 14 cookie consent interfaces (see Appendix A) in a simulated e-commerce environment. We recruited a gender-balanced sample of participants from both the US and the UK. We received 1359 complete responses.
3.1 Consent Interface Design
| Condition Name | Prominence | Location of Definitions | Initial Options | Text Layout | Decision Reversal |
|---|---|---|---|---|---|
| baseline | fully-blocking center panel | preferences panel accessed through show details button | check boxes for all 3 optional cookie types, show details button, allow selection button, accept all cookies button | bulleted | persistent button, no instructions |
| common-banner | non-blocking bottom banner | preferences panel accessed through edit cookie preferences link | edit cookie preferences link, accept all cookies button | paragraph | persistent button, no instructions |
| cornerButton | non-blocking corner button | preferences panel accessed through cookie preferences button | cookie preferences button | N/A | persistent button, no instructions |
| definitions-accordion | fully-blocking center panel | in-line (accordion) | toggles for all 3 optional cookie types, allow selection button, accept all cookies button | bulleted | persistent button, no instructions |
| definitions-inline | fully-blocking center panel | in-line (always visible) | toggles for all 3 optional cookie types, allow selection button, accept all cookies button | bulleted | persistent button, no instructions |
| definitions-sidebanner | fully-blocking side panel | in-line (accordion) | toggles for all 3 optional cookie types, allow selection button, accept all cookies button | bulleted | persistent button, no instructions |
| definitions-tabs | fully-blocking center panel | tabs | tabs for all 4 cookie types, allow selection button, accept all cookies button | bulleted | persistent button, no instructions |
| options-3button | fully-blocking center panel | preferences panel accessed through edit cookie preferences button | edit cookie preferences button, accept only necessary button, accept all cookies button | bulleted | persistent button, no instructions |
| options-3button-banner | non-blocking bottom banner | preferences panel accessed through edit cookie preferences button | edit cookie preferences button, accept only necessary button, accept all cookies button | bulleted | persistent button, no instructions |
| options-button | fully-blocking center panel | preferences panel accessed through edit cookie preferences button | edit cookie preferences button accept all cookies button | bulleted | persistent button, no instructions |
| options-link | fully-blocking center panel | preferences panel access through edit cookie preferences link | edit cookie preferences link, accept all cookies button | bulleted | persistent button, no instructions |
| text-paragraph | fully-blocking center panel | preferences panel accessed through show details button | check boxes for all 3 optional cookie types, show details button, allow selection button, accept all cookies button | paragraph | persistent button, no instructions |
| reversal-cookiePolicy | fully-blocking center panel | preferences panel accessed through show details button | check boxes for all 3 optional cookie types, show details button, allow selection button, accept all cookies button | bulleted | cookie policy, no instructions |
| reversal-Instructions | fully-blocking center panel | preferences panel accessed through show details button | check boxes for all 3 optional cookie types, show details button, allow selection button, accept all cookies button | bulleted | persistent button, instructions |
Table 1: Overview of the 14 cookie consent interface design variants. Design choices that differ from baseline interface are bold.
Figure 1: The boxes below show the positioning of each type of prominence tested in our study.
Figure 2: Example consent interfaces from the study
Table 1 provides an overview of the design variants we evaluated in our user study and Figure 1 shows their positioning within the browser. The variant we labeled baseline serves as the basis of comparison for all other conditions. All of our designs were implemented using the design tool integrated with the OneTrust CMP (version 6.31.0). While the main focus of our study was understanding the impact of users’ location and their device type, we tested 14 cookie consent interface designs in this study. This approach allowed us to ensure that any observed effects were not restricted to a single interface type, and evaluate the usability of both typical and novel designs that varied based on banner prominence, location of cookie category definitions, and which options are shown initially. We tested eight conditions that included a button to access a secondary cookie preference panel with toggles to accept or reject each cookie type (four of these also included check boxes for all optional cookie types in the primary consent interface), two conditions that included a link to access the secondary cookie preference panel, one condition that included a toggle for each cookie type on a separate tab, and three conditions that integrated the cookie preferences panel into the primary consent interface and had no secondary panel. A representative subset of the consent interfaces from the design variants can be seen in Figure 2.
The designs vary along five main design parameters:
• | Prominence: The prominence of the primary (initial) cookie consent interface; this parameter includes the position and shape of the interface (e.g., “center panel,” “bottom banner,” or “side banner”) and whether users are forced to interact with the interface or not (“fully-blocking” or “non-blocking”). | ||||
• | Initial Options: The cookie options that a user can access from the initial interface without clicking through to a secondary cookie preference panel. | ||||
• | Text Layout: The formatting used for the primary cookie consent interface text; the text may be in a single paragraph or a bulleted list. | ||||
• | Decision Reversal: The process for changing or revoking a consent decision; this parameter includes both the presence or absence of instructions on how to reverse a consent decision (“instructions” or “no instructions”) and the location of the option to change a consent decision (“persistent button” or “cookie policy”) | ||||
• | Location of Definitions: Where in the interface the user must go to view definitions of cookie categories; for example, the definitions may be visible when the primary interface appears or the user may have to click a button to see the definitions. | ||||
The first four parameters were identified by Habib et al. [23] as commonly customized in CMP interfaces and potentially affecting usability. In most consent interfaces, term definitions are not present in the initial cookie notice. We opted to vary the location of definitions in order to test a new hypothesis based on the results of Habib et al. [23]. In their study, users in conditions where cookie terms were used but not defined in the initial consent notice were less likely to review the definitions of cookie terms in the preference panel when answering comprehension questions. We hypothesized that adding definitions in the initial notice may help educate users about what cookie terms mean. To test this hypothesis, we added four design variants: definitions-inline, definitions-tabs, definitions-accordion, and definitions-sidebanner. Definitions-inline shows the cookie definitions right below the corresponding toggles in a single fully-blocking panel. Both definitions-accordion and definitions-sidebanner similarly present the definitions with their corresponding toggles; however, each definition is not visible until a plus-sign-shaped button next to the name of the cookie type is pressed. Definitions-sidebanner is identical to definitions-accordion except the panel slides out from the side of the screen rather than appearing at its center. Finally, definitions-tabs is a fully-blocking panel with a tab for each type of cookie containing a definition and toggle.
We also developed two variants to explore the impact of offering thee initial options, similar to designs recommended by noyb [35]. We hypothesize that providing a three-button interface with the option to “Accept all cookies,” “Accept only necessary” cookies, or “Edit cookie preferences” may improve users’ ability to make their preferred decision, as it allows individuals to opt-out of all unnecessary cookies without viewing the complete menu of options available under “Edit cookie preferences.” We evaluate the three-button approach using two variants—options-3button and options-3button-banner. The conditions are identical except that the later presents the buttons in a non-blocking banner rather than a fully-blocking panel.
The other seven design variants we tested were selected based on their potential impact on mobile users as well as common website practices:
• | The text-paragraph condition features interface text formatted in a paragraph rather than a bulleted list, as we hypothesized that mobile users may find the paragraph text more difficult to read. | ||||
• | The options-button and options-link conditions both feature a deceptive design pattern commonly seen in practice: an imbalanced initial options path where users can accept all cookies with a button, but they must access the preference panel in order to make a different decision. | ||||
• | The cornerButton condition features no initial consent notice, requiring users to press the persistent cookie preferences button in order to make a consent decision. As the button takes up more of the screen on mobile phones, we hypothesized that mobile users may be more likely to access the preference interface. | ||||
• | The reversal-cookiePolicy and reversal-Instructions allow us to test variations of the consent decision reversal process. While the GDPR and CCPA both require that users be able to reverse their consent decision, websites often place the option to do so in the cookie policy, with and without instruction on how to do so. | ||||
• | Finally, the common-banner design utilizes a set of design features that are often observed in real world consent interfaces: banner prominence, text formatted in a paragraph, and only a single initial option to accept all cookies, requiring users to click a link to make a different consent decision. | ||||
To facilitate comparison and control for potential confounding effects, the same wording was used across all design variants, with only minor modification of button labels due to design limitations imposed by the OneTrust CMP. All designs except cornerButton included a close button in the form of an "X" that dismisses the banner and accepts the website’s default cookie options. The close button is enabled in most OneTrust layouts by default. All design variants, except for the four definitions variants, included a secondary cookie preferences panel, which was formatted exactly the same as the primary consent interface used in the definitions-inline variant.
The design tools provided by OneTrust placed constraints on our designs. For example, we were unable to implement a condition that displayed definitions via tool tips, which we hypothesized might be convenient for users. In order to create the variants with inline cookie-type definitions, we had to use the templates for the cookie preference interface as cookie banners. This caused the button placement for the consent options in these four variants to differ significantly from the baseline. These differences must be kept in mind when interpreting the results of our study.
Before launching our study, we conducted a pilot study with 72 participants. The results of the pilot were used to refine our survey and ensure that the experiment proceeded as expected; they were not included in our final analysis.
3.2 Data Collection
3.2.1 Participant Recruitment.
We recruited participants using the crowdworking platform Prolific. We performed a power analysis based on our planned statistical tests and determined that 96 participants were required per condition (1350 participants in total). To improve external validity, we separately recruited 450 individuals in each of three gender-balanced age buckets (using Prolific’s “balance sample” feature): 18 to 35 years old, 36 to 53 years old, and over 54 years old. Within each age bucket, participants were selected evenly from the US and the UK. Finally, we recruited 20 additional participants who identified as non-binary, as the balance sample feature does not include such individuals by default.
We chose to balance our sample based on gender and age due to the significant gender and age imbalance in the sample recruited by Habib et al. [23], which was also recruited on Prolific. Moreover, previous research in usable privacy has found effects from age [15, 19] and, to a lesser extent, gender [48]. While other demographic factors (e.g., level of education) may have been useful to balance, we opted to focus on these features.
We posted the recruitment advertisements at different times in the US and UK so that the recruiting period would begin around the end of the traditional work day in each country’s time zones; we began recruiting around 17:00 BST in the UK and 17:00 CDT in the US and recruited until we reached our quota for each bucket.
In order to prevent participants from focusing on the cookie banner, the recruitment advertisement (see Appendix B.1) presented the study as seeking feedback on an e-commerce interface. Potential participants were directed to a screening survey that verified their eligibility and obtained digital informed consent. We assigned eligible individuals randomly to one of 14 conditions. In all conditions, we gave participants the distraction task to add a product to their cart on a simulated e-commerce website called “Cups N’ Such,” similar to the site used by Habib et al. [23], but implemented as a real HTML website rather than an Adobe XD prototype. We told participants that the website was located in the US or UK based on their location. The complete instructions can be seen in Appendix B.2.
Our protocol was approved by Carnegie Mellon University’s Institutional Review Board. While we collected the unique alphanumeric identifiers assigned to participants by Prolific, we did not collect any other personally identifiable information (PII) or any information that would allow us to link Prolific identifiers to PII.
3.2.2 Experimental Procedure.
At the end of the informed consent and screening survey, participants were directed to the version of the Cups N’ Such website that corresponded to their condition. We implemented the Cups N’ Such website on an actual web server on the cupsnsuch.store domain. In addition, we displayed prices in both dollars and British pounds. Participants were presented with the appropriate cookie consent interface as soon as they arrived at the website. Their consent decision was captured by OneTrust and saved to a cookie. The website was additionally instrumented to capture other information about participant interactions with the website. If a participant clicked on an element of the consent interface (e.g., to open the secondary preference interface), a cookie was saved indicating the number of interactions with that element. All cookies and website requests were logged, allowing us to determine the pages viewed by a participant and their consent related behavior (see section 3.3). We did not collect IP addresses. All collected information was associated with a particular participant using their Prolific identifier.
In sum, we directly captured the following data during the task:
(1) | Consent interactions: the button pressed to dismiss the consent interface (if any); the specific cookie options selected (if any); and the number of visits to the cookie preference panel. | ||||
(2) | Information gathering interactions: In the three conditions where the cookie definitions were initially hidden we recorded the number of times participants opened any of the definitions. | ||||
(3) | Website metrics: Time spent on the website, all pages visited on the website, and the participant’s user-agent string. | ||||
In the 11 fully-blocking conditions, once participants made a cookie consent decision or closed the banner, they were freely able to navigate the Cups N’ Such website, including product pages and a cookie policy and privacy policy written for the experiment. Most other links on the website were non-functional. The task was not completed until the participant navigated to the page for a product and clicked “Add to cart.” At this point, they received the link to a post-task survey. In the three non-blocking conditions, participants could interact with the cookie consent interface or ignore it and proceed to the task.
3.2.3 Post-task Survey.
The post-task survey (see Appendix C) had 58 questions (mean time = 14.5 minutes, standard deviation = 8.1 minutes). The survey is based on prior work [23] with added questions to explore effects related to a participant’s country and blocking of cookies, and to probe user sentiment more deeply.
The survey began with questions about participants’ shopping experience and use of cookie settings or browser extensions. The next section asked whether participants remembered making a privacy-related decision and (if so) what that decision was about. They were then asked what cookie consent decision they made (if any, including clicking ’X’ to dismiss the banner). While the responses to this question were not used to analyze user consent decisions, the responses allowed us to ask questions later in the survey based on the decision each participant remembered making. We next asked free-response questions about what participants expected to happen because of their consent decision and the goal of their consent decision.
We added a multiple-choice question that asked participants who reported making no decision to explain why they “decide[d] not to make a selection regarding the use of cookies?” The awareness section concluded with questions about how the participant made their consent decision, how carefully they considered their options, and whether they read the cookie notice text. The final question required participants to rate whether a set of options were available using a Likert scale from “Definitely not available” to “Definitely available.” We summed the number of options that the participant correctly identified as available or not available to calculate a single option recall score.
The next two sections of the survey were designed to evaluate participants’ comprehension of information about cookies as presented in the cookie banners. We first asked participants to answer five factual, multiple-choice questions about the definition of cookies and the cookie categories. The number of correct answers to these questions served as our proxy for participant comprehension. We also added two questions about what happens if they make no decision, as we hypothesized that this may differ between participants living in the US and those living in the UK due to the different legal requirements.2
We then directed participants back to the version of the Cups N’ Such website corresponding to their condition and asked the same seven questions with their previous answers pre-filled. We passed a URL parameter to the website causing all cookies to be cleared and allowing the cookie banner to appear again. We encouraged participants to review the cookie banner and website as they revisited their previous answers. The participants’ first answers to the seven questions provided insights into what information participants in each conditions acquired while interacting (or not interacting) with the consent interface as they typically would, while their revised answers provided insight into how well the consent interfaces conveyed information to users who took the time to review them. To differentiate between the two sets of comprehension questions, the responses prior to returning to the website are referred to as “recall comprehension”; the responses after returning to the website are referred to as “focused comprehension.”
After repeating the comprehension questions, we asked participants what additional options related to cookies (if any) they would like, what option they think the website is recommending, how easy they found the cookie consent interface to understand, and what option they would prefer. This last question included explicit definitions of each cookie type, and it allowed us to evaluate whether participants actually made their preferred decision in interacting with the banner. This section concluded by asking users to rate how easy it was to make their preferred decision.
The next section was largely composed of Likert scale questions designed to gauge participants’ sentiment regarding the cookie consent process and the banner they saw. The first two questions were not present in Habib et al.’s study [23] and focus on participants’ understanding of the legal requirements for cookies in their country. This section also included a new Likert scale question with free-response followup asking participants to compare the cookie consent interface they saw on Cups N’ Such with other consent interfaces they may have seen. Another new question asked participants to rate the extent to which they felt text in the banner was “clear and concise.”
The survey included some additional questions that were included in Habib et al.’s study [23]. However, as they do not relate directly to our research questions we do not discuss them here.
The survey concluded with demographic questions to characterize our sample and evaluate confounding effects. We asked participants to report their age, gender, race/ethnicity, education, and household income. We also asked how frequently they shop online and if they have work experience or education in “a computer-related field, such as computer science or IT.” Participants with computer-related experience were classified as a “tech expert” in our analysis. Due to the inclusion of individuals from the UK, the questions about race/ethnicity, level of education, and household income were modified from those asked by Habib et al. [23] to use language that was more applicable to people from either the US or UK. The wording and specific options present in the gender question were also altered to be more inclusive of non-binary gender identities. Participants could decline to disclose demographic information.
We gave participants a final chance to provide open-ended feedback on the study and then automatically redirected them back to Prolific so that they could be compensated $5.00 for a complete response.
3.3 Data Analysis
We gathered 1375 complete survey responses. 16 of these responses were excluded from analysis as we could not connect them with any website log data.3
3.3.1 Log Data Analysis.
A number of variables of interest had to be derived from the website logs.4 Most significantly, we determined each user’s cookie consent activity solely through the log data collected during the study. While the simplest approach would be to use the consent decision recorded by OneTrust as a proxy for user action, this would obscure nuance within the data. For example, as we configured only strictly necessary cookies to be used by default, a user making no consent decision would appear the same as a person who actively chose to consent only to strictly necessary cookies. Thus, we combined both the OneTrust consent data and button presses recorded via our instrumentation to divide participant action into one of the following categories: accept all, close banner, save preferences with default options, custom selection, and no decision.
Fewer than five participants in each condition who should have been forced to interact with the banner did not have a consent decision recorded by OneTrust or an action recorded via our instrumentation. We suspect their browser configuration may have prevented the banner from appearing or prevented cookies recording their decision from being stored. For example, one participant mentioned using an extension that blocks cookie consent banners entirely. Since we do not have evidence that these participants made a consent decision, they have been included in the no-decision category.
One important metric for any consent interface is how well users are able to make their preferred decision. To account for the different possible assumptions users may make about what happens if they do not make a decision, we analyze user preference data under two different scenarios:
• | Accept Only Necessary Assumption: If a user made no decision or closed the banner, we assume that they would only be opted into strictly necessary cookies. | ||||
• | Accept All Assumption: If a user made no decision or closed the banner, we assume that they would be opted into all cookie categories. | ||||
These assumptions reflect the common practice of websites and the opt-in vs opt-out nature of regulation in each location [18, 37, 53]. For each assumption, we compare the participants’ preferences — as reported in the post-task survey — to the decision they made. We also analyzed the actions and preferences of only the participants who made an active decision, as these data may more accurately reflect the behavior of participants who actively engage with the banner. We excluded the 51 participants who expressed contradictory consent preferences (e.g., a participant who states that they would prefer no cookies but also that they want to accept targeting cookies).
3.3.2 Statistical Analysis.
For each relevant datum derived from the web logs or the post-task survey, we ran statistical tests to evaluate potential explanatory variables. In particular, we independently evaluated whether the datum differed significantly (p ≤ α = 0.05) based on the following explanatory variables: participant condition, country, gender, age, device type,5 and level of tech expertise. For the purpose of analysis, participants were divided into ten-year age buckets (18-27, 28-37, etc.) as well as a “younger” (younger than 38) and an “older” (older than 37) age group. If, in this first set of tests, condition was shown to have a statistically significant effect on the variable, we then ran tests comparing each condition to baseline with respect to the datum of interest. To compare the decisions made in the two three-button conditions with conditions where there is not a third button, clicking “Save Preferences” or “Allow Selection” without selecting any options or pressing a dedicated “Strictly Necessary Cookies Button” are treated as the same action. We also ran statistical tests comparing the effect of conditions that only varied in a single parameter (e.g., definitions-sidebanner and definitions-accordion).
We used R version 4.2.1 to conduct statistical tests. We analyzed categorical data such as actions using Pearson’s chi-squared test or Fisher’s Exact Test.6 We analyzed timing data using a one-way analysis of variance (ANOVA). In section 4 we focus on highlighting statistically significant test results, reporting the test type, p-value, and effect size. For the purpose of brevity, we do not discuss some significant results with small effect sizes relating to demographic characteristics. We used the Bonferoni method to correct all p-values for multiple hypothesis testing. This method is conservative and may lead to more type II errors than alternatives [55].
3.3.3 Thematic Analysis.
We conducted thematic analyses of participants’ responses to six free response questions. One member of the research team served as the primary coder and was responsible for developing and maintaining the codebook, which was based on the themes used by Habib et al. [23]. We used the results of our pilot study (see Section 3.1) to develop a preliminary set of themes for the new question comparing the consent interface to others participants may have seen (question 43).
Once data collection was complete, three members of the research team independently coded 30% of the responses. During this training phase, regular meetings were held with the primary coder to compare results, discuss and reconcile any differences, and implement new codes as necessary. Having reached >80% percent agreement for at least two out of the three coders for all six questions, we divided the remaining data evenly between the three coders such that the researcher with the best reliability for a question coded the remaining 70% of responses. A small number of additional codes were added during this stage of analysis. The primary coder reviewed the first 30% to verify that new codes were appropriately applied across the data set. The completed codebook can be found in Appendix E.
3.4 Limitations
While we have taken steps to recruit a diverse sample we do not claim a representative sample. Our sample is younger and more highly educated than the US and UK general population. Non-white racial and ethnic groups were underrepresented in the US sample [36, 52]. Recent work suggests that results from Prolific are reasonably representative of the US population with regards to questions about privacy and security perceptions and experiences but not knowledge [47]. Additionally, some participants likely behaved differently from their typical behavior due to the knowledge that they were participating in a study. Indeed, a few participants indicated in free response questions that they made their consent decision in order to make sure the website functioned properly for the study. Our qualitative coding is necessarily subjective and influenced by the experience and attitude of the researchers. A different research team may have identified different themes in the data. Finally, while we have explored the effect of several explanatory variables in isolation, they likely have interaction effects as well.
4 RESULTS
In this section, we report the results of our user study. We first summarize participant demographics and then present our findings with respect to each of our three research questions.
4.1 Participant Demographics and Device Configuration
| Gender | Age (Years) | Race/Ethnicity | Education | ||||
|---|---|---|---|---|---|---|---|
| Agender | 0.1% | 18 to 27 | 18.6% | Black | 5.7% | Less than secondary school | 1.5% |
| Female | 48.4% | 28 to 37 | 21.3% | East Asian | 4.1% | Graduated secondary school | 17.4% |
| Genderqueer | 0.2% | 38 to 47 | 18.3% | Hispanic | 4.2% | Some higher education | 24.9% |
| Male | 49.0% | 48 to 57 | 19.3% | Indigenous | 0.9% | Bachelor’s degree | 37.7% |
| Non-binary | 1.6% | 58 to 67 | 17.0% | Middle Eastern | 0.7% | Degree beyond bachelor’s | 18.3% |
| Self-describe | 0.1% | 68 or older | 4.7% | Southeast Asian | 1.8% | Other | 0.3% |
| No response | 0.5% | No response | 0.7% | South Asian | 2.4% | No Response | 0.6% |
| White | 83.4% | ||||||
| Other | 1.1% | ||||||
| No response | 0.9% | ||||||
| Tech Expertise | Cookie Blocking | OS Family | Browser | ||||
| No | 73.7% | No | 82.0% | Windows | 58.8% | Chrome | 59.2% |
| Yes | 26.3% | Yes | 18.0% | Mac OS X | 15.6% | Firefox | 9.0% |
| iOS | 10.3% | Chrome Mobile | 8.5% | ||||
| Android | 9.6% | Mobile Safari | 8.5% | ||||
| Chrome OS | 4.7% | Edge | 6.0% | ||||
| Other Linux | 1% | Safari | 5.6% | ||||
| Other | 3.2% | ||||||
Table 2: Summary of participant demographics collected via the post-task survey. Demographic category names are shortened for space, but the complete text can be found in appendix C.
Table 2 presents an overview of participant demographics (n = 1359). Our sample is well balanced with respect to age and gender. The most common self-reported race or ethnicity was “White or of European descent” with more than 80% of participants selecting this option. Our sample is also skewed towards those with some form of tertiary education: \(56\%\) of our participants had at least a bachelor’s degree or equivalent, with around a third of those individuals reporting some form of graduate education. Our participant pool was roughly evenly divided between individuals located in the US (n=694) and those located in the UK (n=665). The two samples do not differ significantly in participant gender, age, tech expertise, or number of participants who reported blocking cookies. There was a higher proportion of mobile users in the UK sample (\(23.9\%\)) as compared to the US sample (\(12.4\%\)). UK participants in our sample, on average, also reported slightly less education than US participants (\(77.3\%\) UK vs \(83.1\%\) US with at least some higher education, p = 0.0154, Cramer’s V = 0.108).
\(59.2\%\) of participants accessed the task using the desktop version of Google Chrome. The next most common browser was Firefox desktop (\(9.0\%\)). The most common operating system used by our participants was some form of Windows (\(58.8\%\)). The second most common operating system was some version of mac OS X (\(15.6\%\)). \(18.0\%\) of our participants completed the task on a mobile phone, as determined by user-agent string.7 This is a smaller number of mobile users than one would expect [44, 45], likely due to our method of recruitment.8 \(54.3\%\) of mobile users accessed the study from an iOS device while the remaining \(45.7\%\) used an Android device. Mobile participants differed on average from non-mobile participants in a number of categories: mobile users were more likely to identify as female (\(57.6\%\) mobile users vs \(46.41\%\) of non-mobile users were female; chi-squared, p = 0.0322, Cramer’s V = 0.0892); mobile users were more likely to be in a younger age bracket than an older one (\(61.8\%\) of mobile users and \(49.8\%\) percent of non-mobile users were 37 or younger, chi-squared, p = 0.00160, Cramer’s V = 0.133); as previously stated, they were more likely to be located in the UK than the US (\(64.9\%\) of mobile users and \(45.4\%\) of non-mobile users were in the UK; chi-squared, p = 3.51 × 10− 7, Cramer’s V = 0.148); and they were less likely to report blocking cookies (\(11.4\%\) of mobile users and \(19.4\%\) of non-mobile users reported blocking cookies; chi-squared, p = 0.0435; Cramer’s V = 0.0842).
4.2 RQ1: Effect of Country of Residence
We found a number of statistically significant differences between UK and US participants. Most notably, UK-based participants were much less likely to dismiss the initial cookie notice using the close button, and they responded less positively to most sentiment questions about the cookie consent process.
4.2.1 User behavior.
A Pearson’s chi-squared test supports the hypothesis that country of residence affects user consent action (p = 5.28 × 10− 6, Cramer’s V = 0.170). While the effect size is smaller than 0.2, as Figure 3 shows, participants in the UK were more likely to accept all cookies than those in the US. UK participants were also much less likely to dismiss the banner by using the close button than those in the US. One potential explanation may be the relatively higher proportion of participants who used a mobile phone for the task in the UK as compared to the US. However, if we exclude the participants who accessed the task from their phone, the difference between US and UK participants remains statistically significant (chi-squared, p = 1.01 × 10− 5, Cramer’s V = 0.172).
Figure 3: Comparison of consent behavior between participants located in the US and the UK.
Despite this difference in behavior, our participants did not differ significantly in how successful they were in making their preferred decision based on country or any of the other evaluated explanatory variables (see subsection 3.3.2). Across all participants, the most common preferred decision was to “Accept strictly necessary cookies” (\(43.6\%\)). \(24.0\%\) of participants indicated they would prefer some custom combination, with a plurality (42.9) of those participants indicating that they would prefer to enable strictly necessary, functional, and performance cookies. Only \(21.5\%\) of our participants indicated that they would like to “Accept All Cookies.” \(7.06\%\) of the participants wanted no cookies at all, something which is infeasible in an e-commerce environment. \(3.75\%\) of participants selected a self-contradictory preference (i.e., by selecting both that they wanted a certain category of cookies but also wanted no cookies). Finally, \(0.368\%\) of participants selected the “Other” option.
Excluding individuals with contradictory preferences or who would have preferred no cookies, \(50.4\%\) of participants across both countries made their preferred decision, assuming that only strictly necessary cookies were enabled on the website by default. If the website enabled all cookies by default, only \(41.2\%\) of our participants would have made their preferred decision.9 Even if participants who made no decision are excluded, only \(49.1\%\) of the remaining participants successfully made their preferred decision. All of these percentages are within \(5\%\) of the \(45.3\%\) success rate reported by Habib et al. [23]. The most common mistaken decision was to accept all cookies.
In addition to observed differences in consent behavior, UK participants also differed from US participants in how they answered questions about their consent behavior. UK participants were less likely to report that they read the cookie consent notice text (\(25.7\%\) of US vs \(44.9\%\) of UK reported skipping over the notice text; chi-squared, p = 4.47 × 10− 12, Cramer’s V = 0.221). This could reflect a greater degree of habituation to skipping over notice text among UK users. For example, one UK-based participant who reported skipping over the notice text stated: “there’s just too much information - it is unreasonable to expect people to read all the options, so there is a huge risk that most people will just allow all cookies in order to get rid of the annoying pop-up... It’s a bit like T&Cs - most people accept them without reading because they are either too verbose or just gobbledegook.”
4.2.2 Sentiment.
Generally, across the questions intended to evaluate participant sentiment, those in the UK gave fewer positive and more negative answers than their US counterparts. In addition to those questions shown in Figure 4, UK participants also responded less positively when asked to compare the cookie interface they saw in this study to others they may have seen on websites (\(26.4\%\) of UK vs \(36.4\%\) of US rated the interface as much or somewhat better; chi-squared, p = 6.04 × 10− 13, Cramer’s V = 0.233).
Looking at participants’ explanations for their response to the question “Compared to other cookie consent interfaces you may have seen...” there is a large difference between UK and US participants. As can be seen in Table 3, fewer UK participants expressed that the consent interface offered “more choices,” (e.g., “It has more than 2 options”) was “more informative,” (e.g., “There was more information available to explain the choices”) or offered “more clarity” (e.g., “It was much easier to understand”). More UK participants expressed that there was “no difference” (e.g. “It didn’t strike me as being somehow particularly different from the norm.”) between the interface they saw and other consent interfaces. This result likely stems from the difference in the types of consent banners present in the US and UK due to differing regulatory requirements.
| Code | US | UK | Code | US | UK | Code | US | UK |
|---|---|---|---|---|---|---|---|---|
| No Difference | 99 | 145 | More Informative | 169 | 56 | Unhelpful | 61 | 47 |
| More Clarity | 150 | 103 | Less Informative | 4 | 2 | Better Defaults | 15 | 12 |
| Less Clarity | 9 | 8 | Overwhelm with Info/Choice | 4 | 6 | Other | 12 | 6 |
| More Choices | 186 | 44 | Better UX/UI Design | 92 | 42 | Lack of Deceptive Design Patterns | 7 | 9 |
| Preferred Choice Unavailable | 11 | 8 | Worse UX/UI Design | 25 | 11 | Not Sure | 0 | 2 |
Table 3: Frequency of codes for participant responses when asked to explain their rating of the consent interface compared to others they have seen. 665 participants were located in the UK, and 694 participants were located in the US.
Additionally, despite not differing significantly in their success at making their preferred decision, UK-based participants were significantly less likely than US-based participants to say making their preferred decision was “Very easy” (\(26.2\%\) UK vs \(39.8\%\) US participants; chi-squared, p = 2.27 × 10− 5, Cramer’s V = 0.157).
4.2.3 Comprehension.
There was no statistically significant difference in either recall or focused comprehension score between US and UK participants. US and UK participants also did not differ significantly in their ability to recall the available options. Despite this lack of difference in outcome, UK users did respond more negatively to the question “How easy or difficult do you find the cookie consent interface to understand?” (chi-squared, p = 0.000427, Cramer’s V = 0.14133343) \(55.8\%\) and \(65.0\%\) of UK and US users respectively answered that the interface was “Very easy” or “Somewhat easy” to understand. US and UK based participants did differ in their expectations of what would happen if they failed to make a consent decision (p = 0.0482, Cramer’s V = 0.116). UK users were slightly less likely to say that either “Only strictly necessary” cookies would be allowed (\(28.6\%\) of UK participants versus \(29.8\%\) of US participants) or that “No cookies” would be allowed (\(29.6\%\) of UK participants versus \(34.4\%\) of US participants), despite the fact that UK websites are required by law to obtain opt-in consent for cookies other than those strictly necessary. UK participants were more likely to indicate that “All cookies would be allowed” (\(33.7\%\) of UK participants versus \(30.4\%\) of US participants) or that they would be blocked from using the website entirely (\(6.77\%\) of UK participants versus \(5.19\%\) US participants). After participants had the opportunity to review the consent interface again, the difference between the two groups ceased to be statistically significant. While the behavior of real websites varies and is influenced by the relevant regulations, our interface defaulted to only allowing strictly necessary cookies. However, as with most real-world cookie banners we have observed, we provided no indication as to what would happen if a user closed the banner by clicking on the X or just ignored it.
Figure 5: Cookie consent actions for mobile and non-mobile users
As we hypothesized, US and UK participants had different perceptions of the legal requirements in their countries (chi-squared, p = 5.06 × 10− 16, Cramer’s V = 0.259). Under the UK GDPR and ePrivacy Directive, websites can use strictly necessary cookies without permission but must get user permission before using any other cookies [18]. In the US, there is no national law that regulates the use of cookies. Only \(13.2\%\) of UK participants and \(8.93\%\) of US particpiants selected the correct answer for their country. A smaller proportion of UK-based participants stated that they were not sure (\(31.4\%\) of UK vs \(43.7\%\) of US-based participants) or that there were no requirements related to the use of cookies (\(0.451\%\) of UK vs \(8.93\%\) of US-based participants). The most commonly selected perception in both groups was that “Websites must get user permission before using any cookies,” (\(18.5\%\) of UK and \(12.0\%\) of US participants), which is incorrect in both countries. Despite their less positive responses to sentiment questions, on average, UK-based participants reported higher confidence that websites in their country followed applicable law than US-based participants (\(74.0\%\) of UK vs. \(62.5\%\) of US-based participants were “Extremely” or “Moderately” confident; chi-squared, p = 1.86 × 10− 6, Cramer’s V = 0.157).
4.3 RQ2: Effect of Mobile Device Usage
We also found several effects related to mobile device usage. Users on mobile devices were more likely to accept all cookies. Mobile device users also performed worse at answering both recall and focused comprehension questions.
4.3.1 User behavior.
Mobile participants interacted differently with consent banners than non-mobile users across conditions (chi-squared test, p = 4.69 × 10− 5, Cramer’s V = 0.160). While the effect is less than 0.2, as shown in Figure 5, mobile users were more likely to accept all cookies than non-mobile users. They were also slightly more likely to use the accept strictly necessary button than non-mobile users, but a lot less likely to accept strictly necessary using the save preferences or allow selection button.
4.3.2 Comprehension.
Mobile participants performed worse on the recall comprehension questions (Kruskall-Wallis, p = 0.000867, η2 = 0.0103). On average, mobile phone users answered 2.8 questions correctly, while non-mobile users answered 3.16 questions correctly. They also performed worse when answering the comprehension questions after reviewing the consent interface (Kruskall-Wallis, p = 5.18 × 10− 6, η2 = 0.0175), with mobile and non-mobile users answering 3.31 and 3.80 questions correctly on average. Taken as a whole, these results likely reflect that mobile users had a harder time reading and comprehending interface text due to the small screen size. For example, one participant in the definitions-inline condition, when asked to compare the consent interface with others they may have seen, stated: “It was very busy and cluttered” Similarly, in response to the same question, another participant in the reversal-cookiePolicy condition stated: “Too much text, too complicated” Despite their poorer comprehension, mobile users did not respond significantly differently to the question “How easy or difficult do you find the cookie consent interface to understand?”
4.4 RQ3: Effect of Design Parameters
Figure 6: Comparison of consent behavior between consent interface designs. Asterisks indicate conditions where users’ actions were significantly different from the baseline condition.
In this section we discuss the impact of design variants, with a focus on banner prominence, location of cookie category definitions, and initial cookie options. As discussed in subsection 3.2.1, participants were randomly assigned to different consent interface designs at the end of the informed consent process. We found no statistically significant difference between conditions with respect to age, country of residence, gender, tech expertise, mobile phone use, or number of individuals who reported blocking cookies.
Of the considered explanatory variables (see subsection 3.3.2), condition had the largest effect on user consent action (Fisher’s exact, p < 0.001, Cramer’s V = 0.480). Figure 6 shows an overview of the initial consent action broken down by condition. We found significant differences between the baseline condition and the common-banner, cornerButton, definitions-tabs, definitions-sidebanner, options-button, and options-link conditions. The complete pairwise test results are in Appendix D, Table 4. We found no significant difference between conditions in response to any of the eight user sentiment questions.
Banner prominence. We examined differences in user behavior based on banner prominence and found evidence that the side banner resulted in less engagement than a fully-blocking panel in the center of the browser window. However, we did not find evidence of differences in behavior between participants who saw a fully-blocking center panel and those who saw a non-blocking bottom banner. We found support for the hypothesis that the side banner prominence effects user behavior as compared to a fully-blocking cookie notice. The definitions-sidebanner condition varied significantly from the definitions-accordion condition. Participants in definitions-sidebanner were more likely to dismiss the consent notice using the close button and less likely to select only strictly necessary cookies than participants in the definitions-accordion condition. As the main difference between these two conditions is the prominence (they also differ slightly in width and position of the “allow selection” button), this suggests that the notice being placed on the left side of the screen rather than the center likely led to less engagement with the consent process.
Our findings also fail to support the hypothesis that non-blocking bottom banner prominence differs from fully-blocking center prominence. The options-3button-banner and options-3button conditions did not differ significantly from one another. Similarly, the difference in action between the common-banner and options-link conditions10 was not statistically significant.
Location of cookie category definitions. In the three conditions that required users to take an action in order to view the inline cookie category definitions (definitions-accordion, definitions-sidebanner, and definitions-tabs) we collected information about whether participants clicked to view definitions. Only \(10.0\%\) of users in these conditions viewed any definitions. Most of those who accessed the definitions were in the definitions-tabs condition (\(55.2\%\)). This likely results from the unique design of this condition wherein users must click the tab for a cookie category both to view and interact with the toggle for that category and to view the cookie definition. While a plurality of users only accessed a single definition (\(31.0\%\)), one participant in the definitions-tabs condition interacted with the definitions 17 times. \(65.5\%\) of participants who accessed definitions accepted only strictly necessary cookies. Given that few users accessed the definitions, it is not surprising that the presence of definitions in the initial cookie notices seems to have had no affect on user comprehension. This is also true for the definitions-inline condition, where definitions were viewable without clicking, but required some scrolling on most screens. We found no statistically significant difference in number of comprehension questions answered correctly by condition, either before or after revisiting the study website, suggesting that cookie category definitions are largely ignored.
As in the Habib et al. study [23], participants across all groups struggled to correctly pick the definition for functional cookies (\(22.8\%\) before reviewing the interface, \(43.0\%\) after review) and performance cookies (\(46.9\%\) before reviewing the interface, \(67.7\%\) after review). These terms, at least as defined by the ICC UK, are not intuitive and have poor user comprehension. Most participants could recognize the definition of strictly necessary cookies (\(72.0\%\) before reviewing the interface, \(82.1\%\) after review) and targeting cookies (\(78.4\%\) before reviewing the interface, \(86.8\%\)).
Initial cookie options. As shown in Figure 6, participant behavior is similar among most of the conditions that offer an initial option to accept only necessary cookies11 (via a button or through check boxes or toggles) but that participants were more likely to accept all cookies or take no action when doing otherwise required visiting a secondary cookie preferences interface.
Our results suggest that providing an edit cookie preferences button in the initial options is more effective at getting users to engage with the consent process than an edit cookie preferences link. We found a statistically significant difference in action between the options-link and options-button conditions. Participants in the options-button condition were less likely to accept all cookies and more likely to accept only strictly necessary cookies. In both conditions, accepting strictly necessary cookies requires accessing the preference center interface via the button or link, so this result suggests that participants are less likely to view the preference center if its presented via a link rather than a button.
Our findings fail to support the hypothesis that the three-button approach (accept only necessary button, accept all cookies button, and edit cookie preferences button) effects user behavior when compared to offering check boxes for all three optional cookie types. As mentioned in section 3.3.2, to perform this analysis we treated as equivalent clicking “Save Preferences” or “Allow Selection” without selecting any options or pressing a dedicated “Accept Only Necessary” button. Neither of the three-button conditions differed significantly from the baseline in terms of user behavior.
All of the conditions included a close button, with \(16.2\%\) of participants using it to dismiss the initial consent notice. Interestingly, when asked what they expected to happen because of their decision, \(24.0\%\) of those who dismissed the banner with the close button indicated that they expected to receive no cookies (e.g., “I expect that selecting the ’x’ means I do not accept all cookies, and that the site will let me browse for a short period of time until it asks again.”). \(17.2\%\) of those who made the same decision expected the website to enable some or all of the cookies by default (e.g., “For cookies to be collected. Since there was no clear option to disable them I rather just exit out and pretend i’m not being tracked, but I know I am.”). These results reflect the ambiguity of the close button. Indeed, multiple participants requested a way to see what happened if they pressed the close button. For example, one participant in the baseline condition stated that they would like if the interface had “...something to say what will happen if you click x.” Similarly, another participant in the options-3button banner condition stated: “I would like if you click the x if it tells you which cookies will be allowed so i know whats going on if i click it.”
5 DISCUSSION
In this section, we discuss potential reasons behind the differences we observed between US and UK participants and also discuss design implications and recommendations for CMPs. We offer new recommendations regarding the close button and designing for mobile devices, and reiterate recommendations from previous work that are supported by our results.
5.1 US-UK Differences
In our country-to-country comparisons, participants from the UK were more likely to make a consent decision and had lower sentiment towards the consent process. UK participants also expressed greater confidence in their legal protection.
One possible interpretation of the generally lower sentiment of UK participants is related to the relative frequency of cookie banners when browsing in the UK versus the US. Not only have cookie consent notices been in use longer in the UK due to the GDPR and the ePrivacy Directive [18], but UK users are more likely to encounter cookie notices in their daily browsing [14]. This interpretation would suggest that users do not grow fonder of cookie consent interfaces with more exposure. We cannot necessarily rule out other cross-cultural differences as resulting in this effect. One could posit, for example, that UK users are more negative in general than their US counterparts. We find this explanation unlikely, however. Notably, UK and US users did not differ significantly in their response to the non-cookie-related question “How easy or difficult was it to shop on this website?”Despite their lower sentiment about the consent process, UK users were more confident that they had legal protection than US users. This almost certainly reflects the presence of national privacy regulation in the UK that is absent in the US. This finding is consistent with prior literature. Miltgen et al. surveyed younger UK residents and found support for a relationship between perceived regulatory protection and trust of companies and regulators [33]. In the context of cookie consent interfaces, Bellentani found that participants in countries with the GDPR were more willing to disclose personal information than those not under the GDPR [2]. While we found that UK participants could not necessarily identify the requirements of their national regulations with respect to cookies, it seems that the UK GDPR makes them feel more confident in their privacy online.
The different behavior of UK-based participants may reflect habituation to cookie consent notices due to more frequent exposure as well as differences in the types of banners they may have been exposed to due to the GDPR. Websites that comply with the GDPR are incentivized to omit a close button as they must obtain opt-in consent to use any cookies beyond those strictly necessary. We find it likely that UK users have had less exposure to banners with a close button, leading to their lower rate of dismissing or ignoring the banner. The habituation interpretation is further bolstered by the lack of increase in success in making their preferred decision as compared to US-based participants. UK-based participants may have made real consent decisions more often, but it resulted in no more success by this metric.
5.2 Design Implications
We argue that the best solution to the cookie consent problem would be to provide automated mechanisms for consent, reducing user burden. Such an approach is not without its challenges [23]; however, it seems clear that cookie notices are ineffective at allowing users to make informed decisions. Even so, in the short term at least, cookie banners will likely be prevalent. Therefore, we offer recommendations to improve them, informed by our study results.
5.2.1 Consent interfaces should not include a close button without indication of its functionality.
From our results, it is clear that users are currently guessing about what happens when they ignore the cookie banner or click on the X to close it without making a decision. In cases where users assume the close button rejects cookies altogether, this could lead to a privacy violation if cookies are enabled by default. In most cases, a close button is probably unnecessary, but if needed it should be labelled with a phrase such as “Close without accepting optional cookies.”
5.2.2 It should be made clear to users what will happen if they ignore a non-blocking banner.
Notice prominence seems to be a less important factor than the other factors we evaluated. While participant actions in the definitions-sidebanner condition did differ significantly from the baseline and definitions-accordion condition, the actions of participants in the common-banner and options-3button-banner conditions did not differ significantly from their fully-blocking panel counterparts. Some prior work has found effects from prominence [23, 53] while others have not [4]. There is some evidence that a fully-blocking panel may induce slightly more engagement than a non-blocking banner; however, it also may be more disruptive to user activity. Thus, our results suggest that compared to our other recommendations, the position of the banner and whether or not it is fully blocking is unlikely to make a large difference in user behavior. More concerning is that most non-blocking cookie consent interface banners we have observed do not indicate whether or not cookies will be set if a user ignores the banner and does not make a decision. We recommend that non-blocking banners include a prominent statement such as “Only strictly necessary cookies will be enabled unless you make a selection” or “All cookies will be enabled unless you make a selection” to make users aware of what will happen if they ignore the banner.
5.2.3 Cookie consent interfaces should include the ability to make both cookie acceptance and rejection choices directly in the first cookie banner that users see, without requiring users to click to manage preferences.
Similar to prior work [23, 34, 53], we found that the largest effects on participant action came from conditions that required users to navigate to a secondary interface to make a choice other than accept all cookies. To avoid the need for navigating to a secondary interface, websites can provide all cookie choices in the main cookie interface or opt for a solution similar to our three-button approach with an option to accept only strictly necessary cookies. Our study found participants behaved similarly in the baseline and three-button conditions. We also found that users were more likely to click a button than to follow a link to navigate to a secondary interface. Article 7 of the GDPR requires that consent be “freely given,” which entails the option to reject non-essential cookies [27]. Proposed rules from the California Privacy Protection Agency include a similar requirement that options-paths be balanced [10]. When we asked participants for their preferred cookie settings, the third most popular option was to accept all cookies except targeting cookies. This suggests that a four-button option might also be worth exploring, including an “Accept all but advertising” option.
5.2.4 The industry should standardize and adopt more intuitive terms for cookie categories.
We hypothesized that including definitions of cookie categories in the initial interface would aid comprehension; however, it appears that few participants actually read these definitions and we did not observe a significant difference in comprehension based on the presence of definitions. As in prior studies [23], we observed that users struggled to comprehend what cookie terms meant regardless of their condition. Researchers should evaluate alternative terms to identify those that are most intuitive.
5.2.5 Consider consent interface usability on mobile devices.
The cookie consent experience was generally worse for mobile device users. Mobile participants answered fewer recall and focused comprehension questions correctly, suggesting that, similar to privacy policies [43], cookie notices are harder to read and understand on mobile devices. As seen in prior work [4, 46], mobile participants were also less likely to ignore or close the cookie notice, likely due to the relatively larger size of the cookie notice and relatively smaller size of the close button on a mobile device screen. The easiest way for mobile participants to dismiss the banner was to make a selection. Thus, it is even more critical on a mobile device that cookie notices include actionable buttons to accept or reject cookies and that they use succinct and clear terminology.
5.2.6 Recommendations for CMPs.
While utilizing OneTrust made it simple to configure cookie consent interfaces for our study, the default designs recommended by the platform include poor design choices. As of version 202208.1.0, the default “generic,” “GDPR,” and “CCPA” layouts all use a banner design with a close button. While both the “generic” and “GDPR” layouts include a three-button option similar to the one we evaluated, the “CCPA” layout only provides the option to “Accept Cookies,” with no link to the cookie preferences panel. The remaining space in the banner is taken up by a “Do Not Sell My Personal Information” link. We recommend that CMPs not include a close button in their designs by default and that the “CCPA” layout offer at least the same level of choice related to cookies as the other banners.
Some CMPs offer a cookie preferences button in the left or right corner of the page that allows users to revisit their cookie preferences after dismissing the cookie consent interface. CMPs should offer a cookie preference button with a design that makes its function clear. The OneTrust templates include a preference button with a default shield icon with no text identifying the purpose of the button. While the button can be changed to include an icon showing a cookie with a bite taken out of it or text (we implemented it with the text “Cookie Preferences”), configuring this requires that operators write custom code. In general, OneTrust’s tools seem to emphasize obtaining consent to all cookies rather than facilitating an informed decision. In addition to the design features discussed above, the metrics provided to operators by OneTrust’s AB-testing feature focus solely on the number of users consenting to each option. This may contribute to websites using designs that meet minimum compliance requirements but nudge users to accept all cookies. We encourage all CMPs to emphasize usability more in their designs, considering the seven aspects of privacy choice usability identified in the literature [22, 23].
6 CONCLUSION
We conducted a between-subjects research study to answer three main research questions: 1) Do users in the US and the UK interact with or perceive cookie consent interfaces differently? 2) Do users of mobile devices have a less usable experience with cookie consent interfaces as compared to users on computers? 3) How do banner prominence, location of cookie category definitions, and initial cookie options impact attitudes and behavior towards cookie consent interfaces? We recruited a gender- and age-balanced sample of participants on mobile and desktop devices from both the UK and US. We observed lower sentiment towards the consent process among UK-based participants and lower comprehension among mobile users. We also found that the design factor that had the largest effect on participant behavior was the set of initial cookie options included in the cookie banner. In addition, we found that participants had little understanding of what would happen if they clicked the close button or didn’t interact with a non-blocking cookie banner. While our results add more evidence to the case against the notice-and-choice framework for cookie consent, we close by making specific recommendations for website operators and CMPs to improve the usability of cookie consent interfaces.
ACKNOWLEDGMENTS
This research was supported in part by the Innovators Network Foundation, Meta, and the National Science Foundation under grants DGE-2140739 and CNS-2150217. OneTrust provided a complimentary license to use their CMP tools in our study. Any opinions, fndings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily refect the views of the National Science Foundation or other funders.
A CONSENT INTERFACE DESIGN VARIANTS
Figure 7: baseline interface
Figure 8: common-banner interface
Figure 9: cornerButton interface
Figure 10: Clicking the button led to the “Cookie Preferences” page.
Figure 11: definitions-accordion interface
Figure 12: definitions-inline interface
Figure 13: definitions-sidebanner interface
Figure 14: definitions-tabs interface
Figure 15: options-3button interface
Figure 16: options-3button-banner interface
Figure 17: options-button interface
Figure 18: options-link interface
Figure 19: reversal-Instructions interface
Figure 20: reversal-cookiePolicy interface
Figure 21: text-paragraph interface
B RECRUITMENT ADVERTISEMENT AND TASK INSTRUCTIONS
B.1 Recruitment Ad
B.1.1 Feedback on e-commerce interfaces.
• | Reward: $5.00 (approximately $15.00/hr) | ||||
• | Estimated completion time 20 mins. (maximum allowed time: 67 mins.) | ||||
We are inviting you to participate in a voluntary research study evaluating the usability of online interfaces related to e-commerce. Participants must be over the age of 18, reside in the United States, and be fluent in English. You will need to use a tablet, laptop, or desktop computer to participate, as the survey will not display correctly on smartphones or devices with smaller screens.
This survey should take approximately 20 minutes to complete, and participants will be compensated $5.00. During the study, you will be asked to interact with a prototype of a website and answer questions about your experience.
B.1.2 Devices you can use to take this study:
Desktop, Mobile, Tablet
B.2 Task Instructions
Instructions: Please click on the link below to visit a prototype of a website for a new retailer located in the United States/United Kingdom called Cups n’ Such. Please browse the website as you normally would if you were interested in checking out this retailer’s products for the first time and making a purchase. Select a product and put it in your shopping cart. You will then be directed to post-task survey.
Link to prototype website: https: [anonymous]
C POST-TASK SURVEY
The section headings were not visible to participants. Italicized text is used to indicate survey flow and response type. Answer choices are shown in bullets below each question. Answer responses with the text “please specify” or “please describe” included a free response box for participants’ to explain their answer.
C.1 Task Completion
Q1: Please enter your Prolific ID again. (free response field)
Q2: Which country are you located in
• | United States | ||||
• | United Kingdom | ||||
Q3: Were you able to successfully complete the task?
• | Yes, I added a product to my cart | ||||
• | No, I skipped the task | ||||
• | No, I had a technical problem (please describe) | ||||
Q4: Which product did you select?
Question only displayed if participant answered “Yes...” to Q3
• | Delicate Irish Coffee | ||||
• | Insulated Espresso Bubble | ||||
• | Lemon Chiller | ||||
• | Stemware Essentials | ||||
• | The Minimalist | ||||
• | Bamboo Crystal Mug | ||||
• | Hand Painted Tea | ||||
• | Professional Sippy Cup | ||||
• | Grandma’s Diner Special | ||||
• | Shinji | ||||
• | I don’t remember | ||||
Q5: How easy or difficult was it to shop on this website?
• | Very easy | ||||
• | Somewhat easy | ||||
• | Neither easy nor difficult | ||||
• | Somewhat difficult | ||||
• | Very difficult | ||||
Q6: Do you use a browser extension or other tool to block cookies?
• | Yes, the tool blocks some or all cookies | ||||
• | No | ||||
• | I’m not sure | ||||
• | Other (please specify) | ||||
Q7: Have you configured your browser to block cookies?
• | Yes, I have configured my web browser to block some or all cookies | ||||
• | No | ||||
• | I’m not sure | ||||
• | Other (please specify) | ||||
Q8 What is the name of the extension or other tool you use to block cookies? (Free response field)
Question only displayed if participant answered yes to Q6
C.2 Awareness & Needs
Q9 Do you recall making any privacy-related decisions during your interaction with the cups n’ such website?
• | Yes | ||||
• | No | ||||
• | Not sure | ||||
Q10 What was this decision about?
Question only displayed if participant answered yes to Q9
• | The use of cookies on the website | ||||
• | The creation of a username and password for the website | ||||
• | The visibility of credit card info on the website | ||||
• | The use of location data while shopping on the website | ||||
• | Other (please specify) | ||||
Q11 When visiting cups n’ such’s website, you might have seen an interface related to the use of cookies. Which option(s) do you remember selecting? (participants can select multiple options)
• | Accept all cookies | ||||
• | Allow strictly necessary cookies | ||||
• | Allow social media cookies | ||||
• | Allow performance cookies | ||||
• | Allow functional cookies | ||||
• | Allow targeting cookies | ||||
• | “Save preferences” or “Allow selection” without changing any options | ||||
• | Don’t allow any cookies | ||||
• | I clicked the ‘X’ to close the window without selecting any options related to cookies | ||||
• | I didn’t select any options related to the use of cookies | ||||
• | I don’t remember | ||||
Q12 What do you expect to happen since you selected (answer from Q11)? (Free response field)
Question only displayed if participant did not select “I don’t remember,” “I didn’t select any options...” or “I clicked the ‘X’ to close the window...” in Q11
Q13 What do you expect to happen since you clicked the ’x’ without selecting any options related to cookies? (Free response field)
Question only displayed if participant selected “I clicked the ‘X’ to close the window...” in Q11
Q14 What were you trying to achieve when you selected (answer from Q11)? (Free response field)
Question only displayed if participant did not select “I don’t remember,” “I didn’t select any options...” or “I clicked the ‘X’ to close the window...” in Q11
Q15 Why did you decide not to make a selection regarding the use of cookies on the website? (select all that apply) (participants can select multiple options)
Question only displayed if participant selected “I don’t remember,” “I didn’t select any options...” or “I clicked the ‘X’ to close the window...” in Q11
• | I didn’t notice there was a decision to make | ||||
• | I didn’t care what kind of cookies the website was using | ||||
• | I assumed that if I didn’t make a decision the website wouldn’t use cookies at all | ||||
• | I was in a hurry | ||||
• | Other (please specify) | ||||
Q16 Which of the following best describes how you made your decision related to the use of cookies on the cups n’ such website?
Question only displayed if participant did not select “I didn’t select any options...” or “I clicked the ‘X’ to close the window...” in Q11
• | I picked an option based on my actual cookie preferences | ||||
• | I picked whichever option seemed easiest so the consent interface would go away | ||||
• | I picked an option randomly | ||||
• | Other (please specify) | ||||
Q17 How carefully did you consider the options related to cookies on the cups n’ such website?
Question only displayed if participant did not select “I didn’t select any options...” in Q11
• | Not at all carefully | ||||
• | Moderately carefully | ||||
• | Extremely carefully | ||||
Q18 The cookie notice interface included some text. What did you do when you saw it?
Question only displayed if participant did not select “I didn’t select any options...” in Q11
• | Skipped over it | ||||
• | Skimmed it | ||||
• | Read it carefully | ||||
Q19 What options related to cookies do you recall being available to you on this website? (Options for each statement: Definitely not available, Probably not available, Not sure if available, Probably available, Definitely available)
• | Accept all cookies | ||||
• | Allow only strictly necessary cookies | ||||
• | Don’t allow any cookies | ||||
• | Allow social media cookies | ||||
• | Allow performance cookies | ||||
• | Allow functional cookies | ||||
• | Allow targeting cookies | ||||
C.3 Comprehension (recall)
Instructions: Please select the definition that fits best for each of the following terms.
Q20 In the context of the web, what is a cookie?
• | A security token for two-factor authentication | ||||
• | A small piece of data stored on a computer to keep track of information such as logins or websites the user has visited previously | ||||
• | A memorized secret used to confirm the identity of a user | ||||
• | A unique string of numbers separated by periods that identifies each computer using the Internet Protocol to communicate over a network | ||||
• | I don’t know | ||||
Q21 What are strictly necessary cookies?
• | Cookies that are needed for the website to work properly | ||||
• | Cookies that are needed for collecting certain metrics | ||||
• | Cookies that are needed for determining your location | ||||
• | I don’t know | ||||
Q22 What are performance cookies?
• | Cookies that help measure and improve website features | ||||
• | Cookies that are given priority over other cookies on the website | ||||
• | Cookies that make the website run faster | ||||
• | I don’t know | ||||
Q23 What are functional cookies?
• | Cookies that are needed for the website to work properly | ||||
• | Cookies that help personalize the website’s services for you | ||||
• | Cookies that are given lower priority than other cookies on the website | ||||
• | I don’t know | ||||
Q24 What are targeting cookies?
• | Cookies that are used for delivering personalized advertisements | ||||
• | Cookies that help users navigate the website | ||||
• | Cookies that are needed for determining your location | ||||
• | I don’t know | ||||
Q25 Which of the following scenarios do you think are most likely to happen if you do not make a selection regarding the use of cookies, for example, by dismissing the cookie banner by clicking the "x" in the top right corner?
• | I would be blocked from using the website entirely | ||||
• | No cookies would be allowed so the website would not work at all | ||||
• | No cookies would be allowed but the entire website would still work | ||||
• | All cookies would be allowed and the entire website would still work | ||||
• | Only strictly necessary cookies would be allowed but the entire website would still work | ||||
• | Only strictly necessary cookies would be allowed but some parts of the website would still work | ||||
• | No cookies would be allowed but some parts of the website would still work | ||||
Q26 How confident are you in your answer to the previous question?
• | Not at all | ||||
• | Moderately | ||||
• | Extremely | ||||
• | Not sure | ||||
C.4 Comprehension (review)
Instructions: Open the website again in a new tab by clicking the link below and keep it open for the remainder of the survey. (Link to the appropriate website for their condition was present here)
Instructions: Please answer the following questions after you review your options related to cookies.
Q27 You may have seen several cookie options on the prototype website. What additional options related to cookies would you like to have available to you, if any? (free response field)
Instructions: Next, we are going to ask some of questions again with your previous answers marked. After reviewing the information provided about the use of cookies on the website, please edit your answers if you need to.
Instructions: Please select the definition that fits best for each of the following terms.
Q28 In the context of the web, what is a cookie?
Pre-filled with answer from Q20
• | A security token for two-factor authentication | ||||
• | A small piece of data stored on a computer to keep track of information such as logins or websites the user has visited previously | ||||
• | A memorized secret used to confirm the identity of a user | ||||
• | A unique string of numbers separated by periods that identifies each computer using the Internet Protocol to communicate over a network | ||||
• | I don’t know | ||||
Q29 What are strictly necessary cookies?
Pre-filled with answer from Q21
• | Cookies that are needed for the website to work properly | ||||
• | Cookies that are needed for collecting certain metrics | ||||
• | Cookies that are needed for determining your location | ||||
• | I don’t know | ||||
Q30 What are performance cookies?
Pre-filled with answer from Q22
• | Cookies that help measure and improve website features | ||||
• | Cookies that are given priority over other cookies on the website | ||||
• | Cookies that make the website run faster | ||||
• | I don’t know | ||||
Q31 What are functional cookies?
Pre-filled with answer from Q23
• | Cookies that are needed for the website to work properly | ||||
• | Cookies that help personalize the website’s services for you | ||||
• | Cookies that are given lower priority than other cookies on the website | ||||
• | I don’t know | ||||
Q32 What are targeting cookies?
Pre-filled with answer from Q24
• | Cookies that are used for delivering personalized advertisements | ||||
• | Cookies that help users navigate the website | ||||
• | Cookies that are needed for determining your location | ||||
• | I don’t know | ||||
Q33 Which of the following scenarios do you think are most likely to happen if you do not make a selection regarding the use of cookies, for example, by dismissing the cookie banner by clicking the "x" in the top right corner?
Pre-filled with answer from Q25
• | I would be blocked from using the website entirely | ||||
• | No cookies would be allowed so the website would not work at all | ||||
• | No cookies would be allowed but the entire website would still work | ||||
• | All cookies would be allowed and the entire website would still work | ||||
• | Only strictly necessary cookies would be allowed but the entire website would still work | ||||
• | Only strictly necessary cookies would be allowed but some parts of the website would still work | ||||
• | No cookies would be allowed but some parts of the website would still work | ||||
Q34 How confident are you in your answer to the previous question?
Pre-filled with answer from Q26
• | Not at all | ||||
• | Moderately | ||||
• | Extremely | ||||
• | Not sure | ||||
Instructions: Please answer the following questions, referring to the website if necessary.
Q35 What option related to cookies do you think the website is recommending?
• | Accept all cookies | ||||
• | Allow selected cookies | ||||
• | Allow strictly necessary cookies | ||||
• | Other (please specify) | ||||
• | The website isn’t recommending any options (all options are presented equally) | ||||
Q36 How easy or difficult do you find the cookie consent interface to understand?
• | Very easy | ||||
• | Somewhat easy | ||||
• | Neither easy nor difficult | ||||
• | Somewhat difficult | ||||
• | Very difficult | ||||
• | Impossible | ||||
Instructions: For the next question, you may need to refer to the following definitions:
• | Strictly necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies, so this category of cookies cannot be disabled. These cookies do not store any directly identifiable information. | ||||
• | Performance cookies are cookies used specifically for gathering data on how visitors use a website, which pages of a website are visited most often, or if they get error messages on web pages. These cookies monitor only the performance of the site as the user interacts with it. These cookies don’t collect identifiable information on visitors, which means all the data collected is anonymous and only used to improve the functionality of a website. | ||||
• | Functional cookies allow the provision of enhanced functionality and personalization. They may be set by the website or by third-party providers contracted by the website. They are anonymous and don’t track browsing activity across other websites. | ||||
• | Targeting cookies are used to display advertisements that a website or its advertising partners believe are relevant to you and your interests. These cookies may also be used to track your responses to particular ads. These cookies work by uniquely identifying your browser and device. | ||||
Q37 What would be your preferred cookie consent decision for this website? (Select all that apply) (participants can select multiple options)
• | Accept all cookies | ||||
• | Allow strictly necessary cookies | ||||
• | Allow performance cookies | ||||
• | Allow functional cookies | ||||
• | Allow targeting cookies | ||||
• | Don’t allow any cookies | ||||
• | Other (please describe) | ||||
Q38 How easy or difficult would it be for you to make your preferred cookie consent decision?
• | Very easy | ||||
• | Somewhat easy | ||||
• | Neither easy nor difficult | ||||
• | Somewhat difficult | ||||
• | Very difficult | ||||
• | Impossible | ||||
C.5 Sentiment
Q39 Which of the following is required under (answer from Q2) law?
• | Websites must get user permission before using any cookies | ||||
• | Websites can use strictly necessary cookies without permission but must get user permission to use any other cookies | ||||
• | Websites must get user permission before using targeting cookies but can use any other cookies without permission | ||||
• | Websites must give users a choice to decline the use of all cookies | ||||
• | Websites can use strictly necessary cookies but must give users the choice to decline the use of all other cookies | ||||
• | Websites must give users the choice to decline targeting cookies | ||||
• | Websites are not required to get any permissions or offer any choices about cookies | ||||
• | I’m not sure | ||||
Q40 When you visit retail websites in the (answer from Q2), how confident are you that they follow applicable (answer from Q2) laws about cookies?
• | Not at all | ||||
• | Moderately | ||||
• | Extremely | ||||
• | Not sure | ||||
Q41 To what extent do you feel... (Options for each statement: Not at all, Moderately, Extremely, Not sure)
• | Informed about the data being collected by cookies on this website? | ||||
• | Text presented in the interface was clear and concise? | ||||
• | That this cookie consent interface provides the choices you want related to the use of your data? | ||||
• | Informed about your choices related to cookies on this website? | ||||
• | Capable of making a decision related to cookies on this website? | ||||
Q42 Compared to other cookie consent interfaces you may have seen, do you think this cookie consent interface is...
• | Much worse | ||||
• | Somewhat worse | ||||
• | Neither better nor worse | ||||
• | Somewhat Better | ||||
• | Much Better | ||||
• | Not Sure | ||||
• | I have not seen other cookie consent interfaces | ||||
Q43 Why do you feel that this cookie consent interface was (answer from Q42) than other cookie consent interfaces you have seen? (Free response field)
Question only displayed if participant did not selected “I have not seen other cookie consent interfaces” or “Not Sure” in Q42
Instructions: The following questions refer to “your cookie consent decision” which refers to the decision you made about the use of cookies on cups n’ such the first time you encountered the cookie consent interface.
Q44 To what extent do you feel... (Options for each statement: Not at all, Moderately, Extremely, Not sure)
• | Confident that your cookie consent decision was the best option for you? | ||||
• | Comfortable about how data associated with cookies will be used on this website, given your cookie consent decision? | ||||
• | That your cookie consent decision will be honored by the website? | ||||
Question only displayed if participant did not selected “I didn’t select any options related to the use of cookies” in Q11
C.6 Decision Reversal
Instructions: Please refer to the screenshot below for the following questions..
Participants were shown a screenshot of the cups n’ such with the persistent button to change their consent decision.
Q45 Suppose you already made a decision about how cookies can be used on this website. What would you do if you wanted to change your cookie consent decision, or make a decision if you didn’t when first visiting the website? (Free response field)
Q46 What would you do if what you described in your previous answer was not available on the website? (Free response field)
Q47 Did you look at this website’s privacy policy while taking this survey?
• | Yes | ||||
• | No | ||||
• | I don’t remember | ||||
Q48 Did you look at this website’s cookie policy while taking this survey?
• | Yes | ||||
• | No | ||||
• | I don’t remember | ||||
Q49 Did you look at this website’s cookie preference page (with toggles next to cookie categories) while taking this survey?
• | Yes | ||||
• | No | ||||
• | I don’t remember | ||||
C.7 Demographics
Q50 How frequently do you shop online?
• | Never | ||||
• | Less than once a month | ||||
• | A few times a month | ||||
• | A few times a week | ||||
• | Almost every day | ||||
Q51 How frequently do you shop online? What is your age in years? Enter "0" if you prefer not to respond. (Free response field)
Q52 How do you describe your gender identity?
• | Male | ||||
• | Female | ||||
• | Non-binary | ||||
• | Agender | ||||
• | Genderqueer | ||||
• | Prefer to self-describe (Free response field) | ||||
• | Prefer not to respond | ||||
Q53 How do you describe your race or ethnic identity? (You may select more than one option.) (participants can select multiple options)
• | Black or of African descent | ||||
• | East Asian | ||||
• | Hispanic or Latino/a/x | ||||
• | Indigenous (such as Native American, Pacific Islander, or Indigenous Australian) | ||||
• | Middle Eastern | ||||
• | South Asian | ||||
• | Southeast Asian | ||||
• | White or of European descent | ||||
• | Self-describe (Free response field) | ||||
• | Prefer not to respond | ||||
Q54 What is the highest level of school you have completed or the highest degree you have received?
• | Primary school or some secondary school (no high school diploma, GCSE, GED, or equivalent) | ||||
• | Graduated secondary school (high school diploma, GCSE, GED, or equivalent) | ||||
• | Some higher education (less than BA, BS, or equivalent) | ||||
• | Bachelor’s degree (BA, BS, or equivalent) | ||||
• | Additional degree beyond Bachelor’s degree (MA, PhD, or equivalent) | ||||
• | Other (Please specify) | ||||
• | Prefer not to respond | ||||
Q55 What was your approximate household income in 2021 before taxes?
Response options were displayed in £if participant selected “United Kingdom” in Q2 and $ if participant selected “United States” in Q2
• | Less than $10,000 | ||||
• | $10,000 to $19,999 | ||||
• | $20,000 to $29,999 | ||||
• | $30,000 to $39,999 | ||||
• | $40,000 to $49,999 | ||||
• | $50,000 to $59,999 | ||||
• | $60,000 to $69,999 | ||||
• | $70,000 to $79,999 | ||||
• | $80,000 to $89,999 | ||||
• | $90,000 to $99,999 | ||||
• | $100,000 to $149,999 | ||||
• | $150,000 or more | ||||
• | Prefer not to respond | ||||
Q56 Do you have a formal education in a computer-related field, such as computer science or IT? (“Formal education” could mean a completed degree or certificate, or classes or trainings you took towards a degree or certificate.)
• | Yes | ||||
• | No | ||||
Q57 Do you have work experience in a computer-related field, such as computer science or IT?
• | Yes | ||||
• | No | ||||
C.8 Feedback
Q58 If you have any feedback on the survey or cookie consent interface you saw, please leave it here. (Free response field)
D ADDITIONAL STATISTICAL TEST RESULTS
| Success at making | |||||||
|---|---|---|---|---|---|---|---|
| Consent | Site | Considered | Options | preferred decision, | |||
| Action | Recommendation | How You Chose | Options | Available | Accept-all assumption | ||
| Fisher’s Exact | Fisher’s Exact | Fisher’s Exact | Chi-squared | Pairwise Wilcox | Fisher’s Exact | ||
| Condition 1 | Condition 2 | P value | P value | P value | P value | P value | P value |
| baseline | common-banner | < 0.001* | < 0.001* | 1 | 1 | < 0.001* | 1 |
| baseline | cornerButton | < 0.001* | 0.329 | 0.353 | 0.637 | < 0.001* | 0.081 |
| baseline | definitions-accordion | 1 | 0.001* | 1 | 1 | 0.113 | 1 |
| baseline | definitions-inline | 1 | 0.003* | 1 | 1 | 1 | 1 |
| baseline | definitions-sidebanner | 0.014* | 0.010* | 1 | 1 | 0.056 | 1 |
| baseline | definitions-tabs | 0.031* | 0.034* | 1 | 1 | 0.159 | 1 |
| baseline | options-button | < 0.001* | < 0.001* | 1 | 1 | < 0.001* | 0.183 |
| baseline | options-link | < 0.001* | < 0.001* | 1 | 1 | < 0.001* | 1 |
| baseline | options-3button | 1 | < 0.001* | 1 | 1 | 0.406 | 1 |
| baseline | options-3button-banner | 1 | 0.001* | 1 | 1 | 0.003* | 1 |
| baseline | reversal-cookiePolicy | 1 | 0.386 | 1 | 1 | 1 | 1 |
| baseline | reversal-Instructions | 1 | 1 | 1 | 1 | 1 | 1 |
| baseline | text-paragraph | 1 | 1 | 1 | 1 | 1 | 1 |
| definitions-sidebanner | definitions-accordion | 0.004* | 1 | 1 | 0.078 | 1 | 1 |
| definitions-inline | definitions-accordion | 0.743 | 1 | 1 | 0.541 | 1 | 1 |
| definitions-inline | definitions-tabs | 1 | 1 | 1 | 1 | 1 | 1 |
| options-link | options-button | 0.024* | 1 | 1 | 1 | 1 | 1 |
| options-3button-banner | options-3button | 1 | 1 | 1 | 1 | 1 | 0.985 |
| common-banner | options-link | 0.893 | 1 | 1 | 1 | 1 | 1 |
Table 4: Summary of pairwise statistical tests. * indicate a statistically significant difference
| Chi-squared | ||
|---|---|---|
| To what extent do you feel... | P value | Cramer’s V |
| ...informed about the data being collected by cookies on this website? | 1.61 × 10− 2* | 0.104 |
| ...Text presented in the interface was clear and concise? | 8.26 × 10− 2 | N/A |
| ...that this cookie consent interface provides the choices you want related to the use of your data? | 0.000306* | 0.130 |
| ...informed about your choices related to cookies on this website? | 4.52 × 10− 5* | 0.141 |
| ...capable of making a decision related to cookies on this website? | 3.10 × 10− 2* | 0.09923607 |
| ...confident that your cookie consent decision was the best option for you? | 1 | N/A |
| ...comfortable about how data associated with cookies will be used on this website, given your cookie consent decision? | 1 | N/A |
| ...that your cookie consent decision will be honored by the website? | 0.662 | N/A |
| Compared to other cookie consent interfaces you may have seen, do you think this cookie consent interface is... | 6.04 × 10− 13* | 0.233 |
Table 5: US vs UK statistical testing results for “To what extent do you feel...” sentiment questions
E CODEBOOKS
| Code | Description | Example |
|---|---|---|
| More Choices | The participant feels the cookie consent interface was better than others they have seen due to the increased number of choices offered in the interface | “They give you the ability to choose which kind of cookies you want” |
| More Informative | The participant feels the cookie consent interface was better than others they have seen because it offers more information about cookies, data, etc | “This interface explains what the cookies do, so you know what specific ones you need for the website to function.” |
| Less Informative | The participant feels the cookie consent interface was worse than others because it doesn’t offer enough information | “small paragr5, other sites have much more info.” |
| Overwhelmed with Info/Choice | The participant feels that there is too much information or choice provided and is overwhelmed as a result. | “It was comprehensive but too comprehensive, much like many other websites” or “There is too much information to read” |
| Lack of Deceptive Design Patterns | Participant feels that the interface is better than others that they have seen because it doesn’t try to nudge them toward a particular consent choice | “the website wasn’t leaning into suggesting a decision for me” |
| More/Less Clarity | Participant feels that cookie consent interface is better or worse due to the clarity of the text provided within the interface | “It provides more detail about the types of cookies on the actual interface, compared to most websites which require you to open a link and generally give information in more technical jargon.” |
| Better/Worse UX/UI Design | Participant feels the cookie consent interface is better or worse due to user experience or user interface design (e.g. how the banner is presented, how responsive the banner is, etc.) | “Way to jarring, intrusive and doesn’t the user truly engage with it.” |
| Better Defaults | Participant feels that the consent interface was better than others they have seen because the default choices are better | “It defaults to only strictly necessary cookies” |
| Preferred Choice Not Available | Participant feels that the interface is worse than others they have seen as their preferred choice is not available | “There wasn’t a another cookie option” |
| No Difference | Participant expresses that the interface is similar to others they have seen | “looks similar to others” |
| Other | Participant provides an answer which does not fall into the codes above | “Because it went away quickly.” |
| Not Helpful | Participant provides an unintelligible or otherwise irrelevant answer to the question | “:Cookies are yummy:” |
Table 6: Compare Consent Codebook (See Q43 of the post-task survey)
| Code | Description | Example |
|---|---|---|
| Browser: Change Settings | Participant states they would change cookie settings in their browser | “I would go into my browser settings and change it” |
| Browser: Delete History | Participant states they would delete their cookies and/or browser history | “Delete my cookies in my browser’s settings and revisit the website to change my cookie preferences.” |
| Browser Extension | Participant states that they would (or already do) use a browser extension to clear, block, or otherwise control cookies | “I have a browser extension that clears non-whitelisted cookies when the browser is closed so I wouldn’t whitelist the site and I’d get the cookie consent pop-up appear again the next time I visit the site” |
| Cookie Policy | Participant states they would look at the website’s cookie policy | “Click on cookie policy at the bottom.” |
| Cookie Preferences Button | Participant states they would use the “Cookie Preferences” button in bottom right corner of the page | “Click the Cookie Preference button.” |
| Contact Website | Participant states they would contact the website, or use the “Contact” link | “I will contact Cup n’ such customer service.” |
| Give Up | Participants states they would give up, do nothing, leave the settings as they were, or continue shopping on the website anyways | “Just continue using the website as is probably” |
| Leave Website | Participant states they would leave the website or use a different website | “i would just exit the website” |
| Look Through Website | Participants states they would look through different parts of the website (other than the privacy/cookie policy), including settings or FAQs | “I would look for a button that says preferences or settings or something along those lines.” |
| Privacy Policy | Participant states they would look at the website’s cookie policy | “Go to the ‘privacy policy’ link” |
| Private Browsing Mode | Participant states they would use private browsing mode | “I’d open the website again using incognito mode in chrome and go through the cookie selection process again.” |
| Refresh/Revisit | Participants states they would refresh or revisit the website to change their consent decision | “I will close my tab and try to refresh it and than rejoined it.” |
| Search for Info | Participant states they would search for info on how to change their consent decision, either on a search engine or the website | “No idea, if I really wanted to I’d search or google it” |
| Use Different Browser/Device | Participant states they would use a different browser or device to change their consent decision on the website | “revisit the website in another browser.” |
| Not Sure | Participant isn’t sure what they’d do to reverse their consent decision | “Honestly, I don’t know.” |
| Not Helpful | Participant’s response is incomprehensible or not really relevant to the question | “I do not want to change my cookie consent decision” |
| Other: Incorrect | Participant’s response doesn’t fall into the other categories, and isn’t likely to help them change their consent decision | “I’m not sure, maybe hit the back button until options came up?” |
| Other | Participant’s response doesn’t fall into the other categories, but might help them change their consent decision |
Table 7: Decision Reversal Codebook (See Q45 and Q46 of the post-task survey)
| Code | Description | Example |
|---|---|---|
| Ad Targeting | Their consent decision will lead to targeted ads either on or off cups n’ such | “I would see their products on other apps” |
| Enable Website Functionality | Their consent decision will enable only basic functions of the website, i.e. to check out | “That only cookies necessary to allow me to shop on the website will be applied” |
| Their consent decision will let them use some specific functionality of the website or allow them to use all functionality of the website | “I hope the website will store any information about me, like the contents of the shopping cart if I close the window and come back later” | |
| Better Performance | Their consent decision will lead to a better shopping experience on the website, for example with regards to performance | “The website should perform better” |
| Habit | Participant did not state a specific goal, just that they choose their consent decision out of habit | “I don’t know i just always say yes” |
| Continue to Shopping | Their consent decision will let them continue to shopping on the website and/or dismiss the consent notice | “To continue to the site” |
| Default: Accept Cookies | The respondent did not make an actual consent decision and assumes that the site will place some or all cookies (other than strictly necessary) as a result. | “That it would assume I was fine with cookies.” |
| Enable Data Collection | Their consent decision will let the website collect data or will enable some sort of tracking | “That my data will be collected” |
| Less or No Data Collection | Their consent decision will lead to less or no data collection or tracking | “I expect only a little data to be collected by the site.” |
| More Privacy or Security | Their consent decision will lead to increased security or privacy (either generally or something specific) | “I expected my computer to be less at risk for viruses.” |
| Limit Cookies | Participant states that they expected that cookies would be limited without stating another specific goal such as privacy or site functionality. | “Honestly, I have no idea what is entailed pertaining to “strictly necessary” cookies.... I just know I want as few cookies as possible on my device.” |
| Limiting the amount of cookies | ||
| No Cookies | The user expects their consent decision will lead to no cookies being used. | “Accept no cookies” |
| Not Helpful or Ambiguous Response | Participant’s response is incomprehensible or not really relevant to the question. Includes responses where the participant simply repeated that it would enable whatever it was that they selected | “I wanted to see if my favorite cookies were there.” or “it would allow only strictly necessary cookies” |
| Not Sure | Participant isn’t sure what would happen with respect to their consent decision, or said they don’t really understand what cookies are | “Honestly, I don’t know.” |
| Other | Participant’s response doesn’t fall into the above categories |
Table 8: Expectations Codebook (See Q12 and Q13 of the post-task survey)
| Code | Description | Example |
|---|---|---|
| Better Performance | Participant wanted to achieve best performance of the website when shopping | “I’m trying to make sure the site runs as smoothly as possible for me, the consumer & user.” |
| Continue to Shopping | Participant wanted to continue to shopping by dismissing the popup | “To make the pop up go away.” |
| Enable Website Functionality | Participant wanted to allow some specific functionality of the website or use the full functionality of the website | “I was trying to be able to access all sections of the website.” |
| Habit | Participant did not state a specific goal, just that they choose their consent decision out of habit | “I don’t know i just always say yes” |
| Limit Cookies | Participant wanted as few cookies as possible, particularly because they don’t have a good understanding of what cookies are | “It seemed like the safer option because I had no knowledge of the cookies.” |
| Prevent Interruptions | Participant wanted to prevent any future interruptions, errors, or popups when interacting with the website | “I selected allows all cookies so I can get access to the website without any interruptions or pop-ups” |
| Privacy: Limit Collection | Participant wanted to limit the collection and/or use of their data | “Bare minimum private information collected” |
| Privacy: Limit Tracking | Participant wanted to limit tracking (either on or off the website). Includes tracking related to ads | “Prevent tracking across other websites” |
| Privacy: Other | Participant wanted to achieve some other privacy goal, or just generally more privacy | “Protect my privacy as much as possible while still being able to access the website.” |
| Not Sure | Participant isn’t sure what their goal was, or said they don’t really understand what cookies are | “Honestly, I don’t know.” |
| Not Helpful | Participant’s response is incomprehensible or not really relevant to the question | “Did not think think too much about it initially.” |
| Other | Participant’s response doesn’t fall into the above categories |
Table 9: Choice Goals Codebook (See Q14 of the post-task survey)
| Code | Description | Example |
|---|---|---|
| Already Present | Participant’s suggestion is already present on the website | “Allow only necessary cookies” |
| No Cookies | Participant suggested an option to not allow any cookies | “The option to use no cookies” |
| Other Suggestion | Participant has a different suggestion that is not present on the website | “Necessary for Optimum Performance - does not include ad data” |
| Suggested More Info | Participant suggested that the notice could provide additional information related to their consent decision | “Explanation of not accepting any cookies” |
| No Suggestion | Participant is satisfied with the options available, or did not articulate any additional options the website could offer | “I dont know too much about cookies so I am fine with these options” |
| Not Helpful | Participant’s response is incomprehensible or not really relevant to the question | “SUMMER21” |
| Other | Participant’s response doesn’t fall into the other categories |
Table 10: Additional Cookie Options Codebook (See Q27 of the post-task survey)
Footnotes
1 Habib et al. referred to this factor as “options path” and contrasted conditions where all cookie options were displayed in-line on the initial cookie banner with those where the full set of cookie options were only available on a secondary preferences panel. We refer to “initial options” instead to focus on which options are available on the initial banner rather than the path to the full option set.
Footnote2 In the UK websites are prohibited from setting unnecessary cookies until the user has explicitly opted in whereas in the US websites are permitted to set cookies unless a user has opted out. A user who makes no decision has not opted in or opted out.
Footnote3 In most cases, these participants failed to provide their Prolific ID correctly in the initial screening survey. Since the Prolific ID was required to associate the log data with a survey response, this made it impossible to identify their activity on Cups N’ Such.
Footnote4 As participants could revisit the website during portions of the post-task survey, we restricted our task analysis to activity between each participant’s first visit to the website and their first visit to the post-task survey.
Footnote5 In our device type analysis, we grouped together participants on desktop and tablet devices.
Footnote6 Fisher’s exact test was used if more than \(20\%\) of the entries in a given contingency table had less than 5 observations. For tables larger than 2 x 2, a Monte Carlo simulation with 50,000 replications was used to compute the p-value.
Footnote7 We parsed the user-agent string using the user-agents Python package version 2.2.0.
Footnote8 Prolific does not provide statistics about the device configuration of crowdworkers on its platform, however, in a 2021 forum post, the Prolific community manager stated that the majority of participants used desktop devices on the platform [51].
Footnote9 Our analysis assumed that participants who made no decision either received all cookies or no cookies. This is not perfect; some participants made no decision because they liked the defaults they observed in the interface (e.g., one participant stated they did not make a cookie decision because “it looked as though only one type of cookies was checked.”). In reality, it is likely that the options checked by default would be those used if no decision was made; however, in some conditions the checkboxes were not visible without user interaction. The percentage of participants who managed to make their preferred decision if all cookies were enabled by default likely would have been higher than the percentage we calculated. This may serve as a lower-bound.
Footnote10 The common-banner and options-link differ in two ways: notice text and prominence. However, since notice text seems to have little effect on consent action, any difference between these conditions is likely due to prominence.
Footnote11 Among the conditions offering an accept-only-necessary option, only the definitions-sidebanner condition was significantly different from the baseline.
Footnote
Supplemental Material
- Juna Alhumaidi Alharbi, Abdulmohsen Saud Albesher, and Heider Ahmad Wahsheh. 2023. An Empirical Analysis of E-Governments’ Cookie Interfaces in 50 Countries. Sustainability 15, 2 (2023), 23 pages. https://doi.org/10.3390/su15021231Google Scholar
Cross Ref
- Tommaso Maria Bellentani. 2020. The impact of Cookie Consent Notices on user’s privacy concerns: an empirical analysis. Ph. D. Dissertation. Universidade Católica Portuguesa.Google Scholar
- Steven Bellman, Eric J. Johnson, Stephen J. Kobrin, and Gerald L. Lohse. 2004. International Differences in Information Privacy Concerns: A Global Survey of Consumers. The Information Society 20, 5 (2004), 313–324. https://doi.org/10.1080/01972240490507956Google ScholarCross Ref
- Carlos Bermejo Fernandez, Dimitris Chatzopoulos, Dimitrios Papadopoulos, and Pan Hui. 2021. This Website Uses Nudging: MTurk Workers’ Behaviour on Cookie Consent Notices. Proc. ACM Hum.-Comput. Interact. 5, CSCW2, Article 346 (oct 2021), 22 pages. https://doi.org/10.1145/3476087Google ScholarDigital Library
- Dino Bollinger, Karel Kubicek, Carlos Cotrini, and David Basin. 2022. Automating Cookie Consent and GDPR Violation Detection. In USENIX Security Symposium. USENIX, Boston, MA, 2893–2910. https://www.usenix.org/conference/usenixsecurity22/presentation/bollingerGoogle Scholar
- Robert Bond. 2012. The EU e-Privacy directive and consent to cookies. Bus. Law. 68(2012), 215–224.Google Scholar
- Rico Bornschein, Lennard Schmidt, and Erik Maier. 2020. The Effect of Consumers’ Perceived Power and Risk in Digital Information Privacy: The Example of Cookie Notices. Journal of Public Policy & Marketing 39, 2 (2020), 135–154. https://doi.org/10.1177/0743915620902143 arXiv:https://doi.org/10.1177/0743915620902143Google ScholarCross Ref
- Raluca Budiu. 2015. Mobile user experience: Limitations and strengths. https://www.nngroup.com/articles/mobile-ux/Google Scholar
- California Attorney General’s Office. 2019. The California Privacy Rights Act of 2020. https://www.oag.ca.gov/system/files/initiatives/pdfs/19-0021A1%20%28Consumer%20Privacy%20-%20Version%203%29_1.pdf, as of 2023/01/30 15:58:59.Google Scholar
- California Privacy Protection Agency. 2022. California Consumer Privacy Act Regulations. https://cppa.ca.gov/regulations/pdf/20220708_text_proposed_regs.pdf, as of 2023/01/30 15:58:59.Google Scholar
- Hichang Cho, Milagros Rivera-Sánchez, and Sun Sun Lim. 2009. A multinational study on online privacy: global concerns and local responses. New Media & Society 11, 3 (2009), 395–416. https://doi.org/10.1177/1461444808101618Google ScholarCross Ref
- Adrian Dabrowski, Georg Merzdovnik, Johanna Ullrich, Gerald Sendera, and Edgar Weippl. 2019. Measuring Cookies and Web Privacy in a Post-GDPR World. In Passive and Active Measurement. Springer International Publishing, Puerto Varas, Chile, 258–270.Google Scholar
- Martin Degeling, Christine Utz, Christopher Lentzsch, Henry Hosseini, Florian Schaub, and Thorsten Holz. 2019. We Value Your Privacy... Now Take Some Cookies: Measuring the GDPR's Impact on Web Privacy. In Symposium on Network and Distributed System Security(NDSS ’19). ISOC, San Diego, California, USA.Google ScholarCross Ref
- Rob van Eijk, Hadi Asghari, Philipp Winter, and Arvind Narayanan. 2019. The impact of user location on cookie notices (inside and outside of the European Union). In Workshop on Technology and Consumer Protection(ConPro ’19). IEEE, San Francisco, CA.Google Scholar
- Alisa Frik, Leysan Nurgalieva, Julia Bernd, Joyce Lee, Florian Schaub, and Serge Egelman. 2019. Privacy and security threat models and mitigation strategies of older adults. In Fifteenth Symposium on Usable Privacy and Security (SOUPS 2019). USENIX Association, Santa Clara, CA, 21–40.Google Scholar
- Stacia Garlach and Daniel Suthers. 2018. I’m supposed to see that?’AdChoices Usability in the Mobile Environment. In Proceedings of the 51st Hawaii International Conference on System Sciences (Waikoloa, HI, USA). Association for Information Systems, Atlanta, GA, USA, 3779–3788.Google ScholarCross Ref
- GDPR.EU. 2018. General Data Protection Regulation (GDPR). https://gdpr.eu/tag/gdpr/, as of 2023/01/30 15:58:59.Google Scholar
- GDPR.EU. 2022. Cookies, the GDPR, and the ePrivacy Directive. https://gdpr.eu/cookies/, as of 2023/01/30 15:58:59.Google Scholar
- Reza Ghaiumy Anaraky, Kaileigh Angela Byrne, Pamela J. Wisniewski, Xinru Page, and Bart Knijnenburg. 2021. To Disclose or Not to Disclose: Examining the Privacy Decision-Making Processes of Older vs. Younger Adults. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 686, 14 pages. https://doi.org/10.1145/3411764.3445204Google ScholarDigital Library
- Colin M. Gray, Cristiana Santos, Nataliia Bielova, Michael Toth, and Damian Clifford. 2021. Dark Patterns and the Legal Requirements of Consent Banners: An Interaction Criticism Perspective. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems(CHI ’21). Association for Computing Machinery, Virtual Event, Japan, Article 172, 18 pages. https://doi.org/10.1145/3411764.3445779Google ScholarDigital Library
- Johanna Gunawan, Amogh Pradeep, David Choffnes, Woodrow Hartzog, and Christo Wilson. 2021. A Comparative Study of Dark Patterns Across Web and Mobile Modalities. Proc. ACM Hum.-Comput. Interact. 5, CSCW2, Article 377 (oct 2021), 29 pages. https://doi.org/10.1145/3479521Google ScholarDigital Library
- Hana Habib and Lorrie Faith Cranor. 2022. Evaluating the Usability of Privacy Choice Mechanisms. In Eighteenth Symposium on Usable Privacy and Security (SOUPS 2022). USENIX Association, Boston, MA, 273–289. https://www.usenix.org/conference/soups2022/presentation/habibGoogle Scholar
- Hana Habib, Megan Li, Ellie Young, and Lorrie Cranor. 2022. “Okay, Whatever”: An Evaluation of Cookie Consent Interfaces. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems(CHI ’22). Association for Computing Machinery, New Orleans, LA, USA, Article 621, 27 pages. https://doi.org/10.1145/3491102.3501985Google ScholarDigital Library
- Maximilian Hils, Daniel W. Woods, and Rainer Böhme. 2020. Measuring the Emergence of Consent Management on the Web. In Proceedings of the ACM Internet Measurement Conference (Virtual Event, USA) (IMC ’20). Association for Computing Machinery, New York, NY, USA, 317–332. https://doi.org/10.1145/3419394.3423647Google ScholarDigital Library
- IAB Europe. 2022. CMP List. https://iabeurope.eu/cmp-list/, as of 2023/01/30 15:58:59.Google Scholar
- Information Commissioner’s Office. 2022. The UK GDPR. https://ico.org.uk/for-organisations/dp-at-the-end-of-the-transition-period/data-protection-and-the-eu-in-detail/the-uk-gdpr/, as of 2023/01/30 15:58:59.Google Scholar
- Information Commissioner’s Office. 2022. What is valid consent?. https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/consent/what-is-valid-consent/, as of 2023/01/30 15:58:59.Google Scholar
- Natasha Lomas. 2023. EU watchdogs agree on how to handle certain cookie consent dark patterns. https://techcrunch.com/2023/01/20/cookie-taskforce-report-consent-dark-patternsGoogle Scholar
- Eryn Ma and Eleanor Birrell. 2022. Prospective Consent: The Effect of Framing on Cookie Consent Decisions. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems(CHI EA ’22). Association for Computing Machinery, New Orleans, LA, USA, Article 400, 6 pages. https://doi.org/10.1145/3491101.3519687Google ScholarDigital Library
- Dominique Machuletz and Rainer Böhme. 2020. Multiple Purposes, Multiple Problems: A User Study of Consent Dialogs after GDPR. Proceedings on Privacy Enhancing Technologies 2020, 2 (apr 2020), 481–498. https://doi.org/10.2478/popets-2020-0037Google ScholarCross Ref
- Célestin Matte, Nataliia Bielova, and Cristiana Santos. 2020. Do Cookie Banners Respect my Choice? : Measuring Legal Compliance of Banners from IAB Europe’s Transparency and Consent Framework. In IEEE Symposium on Security and Privacy(SP ’20). IEEE, San Francisco, California, USA, 791–809. https://doi.org/10.1109/SP40000.2020.00076Google ScholarCross Ref
- Maryam Mehrnezhad, Kovila Coopamootoo, and Ehsan Toreini. 2022. How Can and Would People Protect From Online Tracking?Proceedings on Privacy Enhancing Technologies 1 (2022), 105–125. http://dro.dur.ac.uk/34795/Google Scholar
- Caroline Lancelot Miltgen and H. Jeff Smith. 2015. Exploring information privacy regulation, risks, trust, and behavior. Information & Management 52, 6 (2015), 741–759. https://doi.org/10.1016/j.im.2015.06.006Google ScholarDigital Library
- Midas Nouwens, Ilaria Liccardi, Michael Veale, David Karger, and Lalana Kagal. 2020. Dark Patterns after the GDPR: Scraping Consent Pop-Ups and Demonstrating Their Influence. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems(CHI ’20). Association for Computing Machinery, Honolulu, HI, USA, 1–13. https://doi.org/10.1145/3313831.3376321Google ScholarDigital Library
- noyb. 2022. Cookie Banners. https://noyb.eu/en/project/cookie-banners, as of 2023/01/30 15:58:59.Google Scholar
- Office for National Statistics. 2021. Population estimates by ethnic group and religion, England and Wales: 2019. https://www.ons.gov.uk/peoplepopulationandcommunity/populationandmigration/populationestimates/articles/populationestimatesbyethnicgroupandreligionenglandandwales/2019, as of 2023/01/30 15:58:59.Google Scholar
- Office of the California Attorney General. 2018. California Consumer Privacy Act of 2018. https://leginfo.legislature.ca.gov/faces/codes_displayText.xhtml?division=3.&part=4.&lawCode=CIV&title=1.81.5, as of 2023/01/30 15:58:59.Google Scholar
- Emmanouil Papadogiannakis, Panagiotis Papadopoulos, Nicolas Kourtellis, and Evangelos P. Markatos. 2021. User Tracking in the Post-Cookie Era: How Websites Bypass GDPR Consent to Track Users. In Proceedings of the Web Conference 2021(WWW ’21). Association for Computing Machinery, Ljubljana, Slovenia, 2130–2141. https://doi.org/10.1145/3442381.3450056Google ScholarDigital Library
- Iskander Sanchez-Rola, Matteo Dell’Amico, Platon Kotzias, Davide Balzarotti, Leyla Bilge, Pierre-Antoine Vervier, and Igor Santos. 2019. Can I Opt Out Yet? GDPR and the Global Illusion of Cookie Control. In Proceedings of the 2019 ACM Asia Conference on Computer and Communications Security(Asia CCS ’19). Association for Computing Machinery, Auckland, New Zealand, 340–351. https://doi.org/10.1145/3321705.3329806Google ScholarDigital Library
- Cristiana Santos, Nataliia Bielova, and Célestin Matte. 2020. Are cookie banners indeed compliant with the law?Technology and Regulation 2020 (2020), 91–135.Google Scholar
- Cristiana Santos, Arianna Rossi, Lorena Sanchez Chamorro, Kerstin Bongard-Blanchy, and Ruba Abu-Salma. 2021. Cookie Banners, What’s the Purpose? Analyzing Cookie Banner Text Through a Legal Lens. In Proceedings of the 20th Workshop on Workshop on Privacy in the Electronic Society(WPES ’21). Association for Computing Machinery, Virtual Event, Republic of Korea, 187–194. https://doi.org/10.1145/3463676.3485611Google ScholarDigital Library
- Ashutosh Kumar Singh, Nisarg Upadhyaya, Arka Seth, Xuehui Hu, Nishanth Sastry, and Mainack Mondal. 2016. What Cookie Consent Notices Do Users Prefer: A Study In The Wild. In European Workshop on Usable Security(EuroUSEC ’22). ISOC, Karlsruhe, Germany.Google Scholar
- Ravi Inder Singh, Manasa Sumeeth, and James Miller. 2011. Evaluating the readability of privacy policies in mobile environments. International Journal of Mobile Human Computer Interaction (IJMHCI) 3, 1(2011), 55–78.Google ScholarDigital Library
- StatCounter. 2022. Desktop vs Mobile vs Tablet Market Share United Kingdom. https://gs.statcounter.com/platform-market-share/desktop-mobile-tablet/united-kingdom, as of 2023/01/30 15:58:59.Google Scholar
- StatCounter. 2022. Desktop vs Mobile vs Tablet Market Share United States Of America. https://gs.statcounter.com/platform-market-share/desktop-mobile-tablet/united-states-of-america, as of 2023/01/30 15:58:59.Google Scholar
- Emmanuel Syrmoudis, Stefan Mager, Sophie Kuebler-Wachendorff, Paul Pizzinini, Jens Grossklags, and Johann Kranz. 2021. Data Portability between Online Services: An Empirical Analysis on the Effectiveness of GDPR Art. 20.. In Privacy Enhancing Technologies Symposium(PETS ’21, Vol. 2021). Sciendo, Virtual Conference, 351–372. Issue 3.Google ScholarCross Ref
- Jenny Tang, Eleanor Birrell, and Ada Lerner. 2022. Replication: How Well Do My Results Generalize Now? The External Validity of Online Privacy and Security Surveys. In Eighteenth Symposium on Usable Privacy and Security (SOUPS 2022). USENIX Association, Boston, MA, 367–385. https://www.usenix.org/conference/soups2022/presentation/tangGoogle Scholar
- Sigal Tifferet. 2019. Gender differences in privacy tendencies on social network sites: A meta-analysis. Computers in Human Behavior 93 (2019), 1–12. https://doi.org/10.1016/j.chb.2018.11.046Google ScholarDigital Library
- Michael Toth, Nataliia Bielova, and Vincent Roca. 2022. On dark patterns and manipulation of website publishers by CMPs. In Proceedings on Privacy Enhancing Technologies(PETS ’22). Sciendo, Sydney, Australia, 478–497.Google ScholarCross Ref
- Sabine Trepte, Leonard Reinecke, Nicole B. Ellison, Oliver Quiring, Mike Z. Yao, and Marc Ziegele. 2017. A Cross-Cultural Perspective on the Privacy Calculus. Social Media + Society 3, 1 (2017), 2056305116688035. https://doi.org/10.1177/2056305116688035 arXiv:https://doi.org/10.1177/2056305116688035Google ScholarCross Ref
- Josh Tulloch. 2021. Mobile Phone Responses. https://community.prolific.co/t/mobile-phone-responses/237 Retrieved on December 8th, 2021.Google Scholar
- U.S. Census Bureau. 2021. QuickFacts. https://www.census.gov/quickfacts/fact/table/US/PST045221, as of 2023/01/30 15:58:59.Google Scholar
- Christine Utz, Martin Degeling, Sascha Fahl, Florian Schaub, and Thorsten Holz. 2019. (Un)Informed Consent: Studying GDPR Consent Notices in the Field. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security(CCS ’19). Association for Computing Machinery, London, United Kingdom, 973–990. https://doi.org/10.1145/3319535.3354212Google ScholarDigital Library
- Rob van Eijk, Hadi Asghari, Philipp Winter, and Arvind Narayanan. 2021. The Impact of User Location on Cookie Notices (Inside and Outside of the European Union). https://doi.org/10.48550/ARXIV.2110.09832Google ScholarCross Ref
- Tyler J VanderWeele and Maya B Mathur. 2019. Some desirable properties of the Bonferroni correction: is the Bonferroni correction really so bad?American Journal of Epidemiology 188, 3 (2019), 617–618.Google Scholar
- Jacob O Wobbrock. 2006. The future of mobile device research in HCI. In CHI Workshop Proceedings: What is the Next Generation of Human-Computer Interaction? (Montréal, Québec, Canada). Association for Computing Machinery, New York, NY, USA, 131–134.Google Scholar
Index Terms
- A US-UK Usability Evaluation of Consent Management Platform Cookie Consent Interface Design on Desktop and Mobile
Recommendations
Dark Patterns after the GDPR: Scraping Consent Pop-ups and Demonstrating their Influence
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing SystemsNew consent management platforms (CMPs) have been introduced to the web to conform with the EU's General Data Protection Regulation, particularly its requirements for consent when companies collect and process users' personal data. This work analyses how ...
“Okay, whatever”: An Evaluation of Cookie Consent Interfaces
CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing SystemsMany websites have added cookie consent interfaces to meet regulatory consent requirements. While prior work has demonstrated that they often use dark patterns — design techniques that lead users to less privacy-protective options — other usability ...
My Eyes Don't Consent! Exploring Visual Attention in Cookie Consent Interfaces
ETRA '24: Proceedings of the 2024 Symposium on Eye Tracking Research and ApplicationsEach cookie consent interface has a different design variant, featuring various levels of information details, which makes the interaction highly challenging. This study aimed to explore how users' visual attention differs across different variants of ...
Comments