unsealing knowledge

Semantic Scholar shared a short paper ("Sealed Knowledges: A Critical Approach to the Usage of LLMs as Search Engines") to my "Research Feed", with this as their automated TLDR:

It is proposed that doubting the outputs of LLMs can function as a feminist intervention that resists the marginalization and sealing of certain knowledges and perspectives through the usage of LLMS as chatbots.

@lindemann2023sealed_paper:

This research examines the implications of the usage of large language models (LLMs) as search engines on knowledge. Drawing on feminist theories of knowledge, I argue that LLMs used to generate direct answers to search engine inquiries both rely on and reinforce a disembodied and non-situated view of knowledge. This, it is argued, can lead to a "sealing" of non-dominant knowledges. Through this sealing of knowledges, marginalized voices may be heard even less than before. Lastly, drawing on the works of feminist theorists such as Donna Haraway and Sara Ahmed, the research proposes that doubting the outputs of LLMs can function as a feminist intervention that resists the marginalization and sealing of certain knowledges and perspectives through the usage of LLMS as chatbots. This research as part of a wider discourse on the usage of LLMs as search engines is crucial considering the current trend of major search engine providers to integrate LLMs for the production of direct answers into their search engines.

I found the author, PhD student Nora Freya Lindemann (ResearchGate | Twitter), sharing a poster version [@lindemann2023sealed_poster] on Twitter.

I found the paper and poster very interesting—a provocative feminist engagement with @shah2022situating—, and shared the following reply:

@danielsgriffin via Twitter on Aug 29, 2023

This is great! I’m really curious to read what you find in how different platforms & tools \*may\* make strides to support \*unsealing knowledge\*, whether through articulations that help users doubt & dig deeper, providing multiple drafts, or RAG adaptations by marginalized voices?

term: sealed

The 'sealing'/'sealed' language comes from the German "Versiegelte Oberflächen" in @mühlhoff2018digitale.

See also: seams, seamless, and seamful design in @eslami2016first

See also work from Kelley Cotter ([website](https://kelleycotter.com/) | [Twitter](https://twitter.com/kelleyhastwoes)):

"“Shadowbanning is not a thing”: black box gaslighting and the power to independently know and credibly critique algorithms" [-@cotter2021shadowbanning]
"Practical knowledge of algorithms: The case of BreadTube" [-@cotter2022practical]

See also, @eyert2022extending [pp. 30--31], emphasis added:

The second property of representation is variation in accountability, transparency, explainability, and the centralization of information, which we summarize under the term opacity. There are two types: sociomaterial opacity arises through the concentration of massive data sets in the hands of a few private companies (Fourcade & Healy 2017a) or other organizations, through centralized decisions about feature selection and categorizations, or through the inaccessibility of closed-source algorithms. Epistemic opacity (Humphreys 2004), by contrast, is the inherent methodological intransparency of approaches like machine learning or computer simulation.[15] Sociomaterial opacity has often been a source of conflict, for instance where users on social media platforms dispute the validity of inferred categories, such as sexual orientation or health status. Reversely, the deliberate establishment and maintenance of opacity has been brought forward as a value and right in itself (programmatically Glissant 1997; with respect to algorithmic systems, see Ananny & Crawford 2018) and demanded by data subjects. For instance, the intensive debate about privacy and data protection discusses what kinds of data can legitimately be gathered, combined, and re-used and under what conditions (Bayamlioglu 2021; Kosta 2022). In many cases, representation practices have triggered resistance, but if challenging the epistemic assumptions of organizations requires yet more epistemic tools and authority to be mobilized, “[t]he cost of disagreeing” (Latour 1990a, p. 34) might increase. Whether asymmetry and opacity will lessen trust in regulation or rather habituate users to interacting with the **“sealed surfaces” of user interfaces (Mühlhoff 2018**; own translation) is an open empirical question.
___
15. The difference between sociomaterial and epistemic opacity becomes tangible in recent debates around attempts to create “explainable AI,” which address the problem that AI systems are often incomprehensible despite complete access to their internal procedures.