index.xml

<?xml version="1.0" encoding="UTF-8"?>
<rss  xmlns:atom="http://www.w3.org/2005/Atom" 
      xmlns:media="http://search.yahoo.com/mrss/" 
      xmlns:content="http://purl.org/rss/1.0/modules/content/" 
      xmlns:dc="http://purl.org/dc/elements/1.1/" 
      version="2.0">
<channel>
<title>The KBLab Blog</title>
<link>https://kb-labb.github.io/</link>
<atom:link href="https://kb-labb.github.io/index.xml" rel="self" type="application/rss+xml"/>
<description></description>
<generator>quarto-1.6.42</generator>
<lastBuildDate>Thu, 06 Mar 2025 23:00:00 GMT</lastBuildDate>
<item>
  <title>Welcome KB-Whisper, a new fine-tuned Swedish Whisper model!</title>
  <dc:creator>Leonora Vesterbacka</dc:creator>
  <dc:creator>Faton Rekathati</dc:creator>
  <dc:creator>Robin Kurtz</dc:creator>
  <dc:creator>Justyna Sikora</dc:creator>
  <dc:creator>Agnes Toftgård</dc:creator>
  <link>https://kb-labb.github.io/posts/2025-03-07-welcome-KB-Whisper/</link>
  <description><![CDATA[ 





<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://kb-labb.github.io/posts/2025-03-07-welcome-KB-Whisper/images/KBLab-whisper.jpg" class="img-fluid figure-img" style="width:300.0%"></p>
<figcaption>The team behind KB-Whisper. Back row: Agnes Toftgård, Robin Kurtz, Justyna Sikora. Front row: Leonora Vesterbacka, Faton Rekathati. Photography: Lina Löfström Baker/KB</figcaption>
</figure>
</div>
<section id="improving-swedish-speech-recognition" class="level2">
<h2 class="anchored" data-anchor-id="improving-swedish-speech-recognition">Improving Swedish speech recognition</h2>
<p>KBLab proudly presents KB-Whisper, a speech to text model fine-tuned using 50,000 hours of transcribed speech. Traditionally, Automatic Speech Recognition (ASR) systems have been based on models that either require an extensive unsupervised pretraining or supervised training that demands very high-quality orthographic transcripts, which are rare and expensive to produce.</p>
<p>The Whisper model <span class="citation" data-cites="radford2022robustspeechrecognitionlargescale">(Radford et al. 2022)</span>, originally released by OpenAI has revolutionized automatic speech recognition, by showing that high performance could be achieved with a slight decrease in quality of the transcript, thus unlocking large amounts of training data that has hitherto not been used. Subtitles for TV often use abbreviations to fit the text on the screen, and are not considered a gold standard transcription. However, <span class="citation" data-cites="radford2022robustspeechrecognitionlargescale">Radford et al. (2022)</span> showed that this training data, extracted from the web, was still good enough for Whisper to learn.</p>
<p>The massive improvement gain with Whisper has been shown for English, and this result is directly proportional to the amount of English training data available on the web. For languages with fewer speakers, this type of data is less represented on the web and leads to poorer performance. In order to bridge this performance gap, the team at KBLab have constructed a training dataset of transcribed Swedish speech of unprecedented size, which is used to fine-tune Whisper models.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://kb-labb.github.io/posts/2025-03-07-welcome-KB-Whisper/images/KBLab-team.jpg" class="img-fluid figure-img" style="width:300.0%"></p>
<figcaption>The team behind KB-Whisper. Back row: Agnes Toftgård, Robin Kurtz, Justyna Sikora. Front row: Leonora Vesterbacka, Faton Rekathati. Photography: Lina Löfström Baker/KB</figcaption>
</figure>
</div>
</section>
<section id="subtitles-parliament-recordings-and-dialect-archives" class="level2">
<h2 class="anchored" data-anchor-id="subtitles-parliament-recordings-and-dialect-archives">Subtitles, Parliament recordings and dialect archives</h2>
<p>The National Library of Sweden is responsible for collecting, preserving and giving access to everything that is published in Sweden. The collections include the audiovisual archives that hold TV broadcasted in Sweden. Swedish subtitles paired with spoken Swedish from TV broadcasts constitute a large portion of the training data. Parliament recordings paired with high quality transcripts in the form of protocols provide the second largest source of the training data. <a href="https://huggingface.co/datasets/KBLab/rixvox-v2">This dataset</a> is made publicly available on Huggingface, and constitute 23,000 hours of transcribed Swedish speech.</p>
<p>Both of these data sources have the advantage of covering wide variations of spoken Swedish. In order to enhance KB-Whisper’s performance in transcribing rare variations of Swedish, dialect recordings from The Institute for language and folklore (Isof) are included. Subtitles are also extracted from YouTube channels with Swedish content. Finally, datasets collected as crowd sourced initiatives such as Mozillas CommonVoice, Googles FLEURS and the Nordic Speech Technology (NST) dataset are used partly in the training, and partly in the evaluation.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://kb-labb.github.io/posts/2025-03-07-welcome-KB-Whisper/images/bandrulle.jpg" class="img-fluid figure-img" style="width:300.0%"></p>
<figcaption>From the audiovisual archives of the National Library of Sweden. Photography: Lina Löfström Baker/KB</figcaption>
</figure>
</div>
</section>
<section id="two-stages-of-data-quality" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="two-stages-of-data-quality">Two stages of data quality</h2>
<p>To assess the quality of transcriptions, i.e.&nbsp;how well the subtitles, protocols and other transcriptions match the spoken audio, we implemented a preprocessing pipeline. The training examples are split into small 30-seconds chunks and each audio chunk is transcribed using OpenAI’s Whisper-large-v3 and KBLabs VoxRex <span class="citation" data-cites="wav2vec2">(Malmsten, Haffenden, and Börjeson 2022)</span>. Then the overlap between the original transcript and the two AI-generated transcriptions are assessed using Character Error Rate (CER), BLEU and ROUGE scores.</p>
<p>The first quality assessment catches examples with low or no overlap, but still of sufficient quality for the model to learn from, yielding a large training dataset with an increased probability of covering rare Swedish words and names, denoted below as the Stage 1 data. The second quality assessment focuses on defining the style of transcription, aiming to teach the model how to transcribe rather than providing a large number of examples. Two styles of transcriptions are defined: one more subtitle-like (Stage 2-subtitle), and one more orthographic (Stage 2-standard) for more precise transcription.</p>
<p>Each stage is defined by a set of criteria on CER, BLEU and ROUGE, which are outlined in Table 1.</p>
<div class="column-body">
<table class="caption-top table">
<thead>
<tr class="header">
<th></th>
<th>BLEU</th>
<th>CER-head</th>
<th>CER-tail</th>
<th>ROUGE</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Stage 1</td>
<td>&gt; 0.2</td>
<td>-</td>
<td>-</td>
<td>-</td>
</tr>
<tr class="even">
<td>Stage 2 standard</td>
<td>&gt; 0.6</td>
<td>&lt; 0.3</td>
<td>&lt; 0.3</td>
<td>&gt; 0.7</td>
</tr>
<tr class="odd">
<td>Stage 2 subtitle</td>
<td>&gt; 0.6</td>
<td>&lt; 0.4</td>
<td>&lt; 0.4</td>
<td>-</td>
</tr>
</tbody>
</table>
</div>
<p>The resulting hours the fall into each category is presented in Table 2.</p>
<div class="column-body">
<table class="caption-top table">
<colgroup>
<col style="width: 18%">
<col style="width: 21%">
<col style="width: 30%">
<col style="width: 29%">
</colgroup>
<thead>
<tr class="header">
<th style="text-align: right;">Dataset</th>
<th style="text-align: right;">Stage 1 (h)</th>
<th style="text-align: right;">Stage 2 standard (h)</th>
<th style="text-align: right;">Stage 2 subtitle (h)</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: right;">Subtitles</td>
<td style="text-align: right;">34,261</td>
<td style="text-align: right;">3,110</td>
<td style="text-align: right;">6,928</td>
</tr>
<tr class="even">
<td style="text-align: right;">Riksdag</td>
<td style="text-align: right;">21,949</td>
<td style="text-align: right;">5,119</td>
<td style="text-align: right;">8,710</td>
</tr>
<tr class="odd">
<td style="text-align: right;">ISOF</td>
<td style="text-align: right;">54</td>
<td style="text-align: right;">54</td>
<td style="text-align: right;">54</td>
</tr>
<tr class="even">
<td style="text-align: right;">NST</td>
<td style="text-align: right;">250</td>
<td style="text-align: right;">250</td>
<td style="text-align: right;">250</td>
</tr>
<tr class="odd">
<td style="text-align: right;"><strong>Total</strong></td>
<td style="text-align: right;"><strong>56,514</strong></td>
<td style="text-align: right;"><strong>8,533</strong></td>
<td style="text-align: right;"><strong>15,942</strong></td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="based-on-open-weights-from-openais-whisper" class="level2">
<h2 class="anchored" data-anchor-id="based-on-open-weights-from-openais-whisper">Based on open weights from OpenAI’s Whisper</h2>
<p>Following the excellent Whisper fine-tuning tutorial from <a href="https://github.com/huggingface/community-events/tree/main/whisper-fine-tuning-event">Huggingface</a> we fine-tune all sizes of Whisper models on our Swedish training dataset of unprecedented size. The training is performed in a two-stage approach, where the first stage leverages the large Stage 1 dataset, followed by two parallel training stages where the model is either trained on the Stage 2-subtitle data or the Stage 2-standard data.</p>
<p>The training is executed on the Leonardo Supercomputer hosted by CINECA (Italy), that we were granted access to through a <a href="https://eurohpc-ju.europa.eu/index_en">EuroHPC JU</a> AI and data-intensive applications call.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://kb-labb.github.io/posts/2025-03-07-welcome-KB-Whisper/images/leonardo.png" class="img-fluid figure-img" style="width:300.0%"></p>
<figcaption>The Leonardo Supercomputer. Source <a href="https://leonardo-supercomputer.cineca.eu/">CINECA</a></figcaption>
</figure>
</div>
</section>
<section id="a-great-improvement-in-swedish-asr" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="a-great-improvement-in-swedish-asr">A great improvement in Swedish ASR</h2>
<p>We have evaluated the models on three datasets: FLEURS (train and test set), NST (test set), and Common Voice 16.0 (train, validation, and test set). The CommonVoice and FLEURS data has not been part of the training set and can therefore serve as a benchmark for the models’ out-of-domain performance.</p>
<p>To compare our newly trained models with OpenAI’s models, we calculate Word Error Rate (WER) and BLEU scores for each of the mentioned datasets. WER measures transcription accuracy by calculating the percentage of words that are substituted, deleted, or inserted, while the BLEU score evaluates how well a transcription matches the reference text.</p>
<p>The results evaluated in terms of WER is presented in the table below.</p>
<div class="column-body">
<table class="caption-top table">
<thead>
<tr class="header">
<th>Model size</th>
<th></th>
<th>FLEURS</th>
<th>CommonVoice</th>
<th>NST</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><a href="https://huggingface.co/KBLab/kb-whisper-tiny">tiny</a></td>
<td><strong>KBLab</strong></td>
<td><strong>13.2</strong></td>
<td><strong>12.9</strong></td>
<td><strong>11.2</strong></td>
</tr>
<tr class="even">
<td></td>
<td>OpenAI</td>
<td>59.2</td>
<td>67.8</td>
<td>85.2</td>
</tr>
<tr class="odd">
<td><a href="https://huggingface.co/KBLab/kb-whisper-base">base</a></td>
<td><strong>KBLab</strong></td>
<td><strong>9.1</strong></td>
<td><strong>8.7</strong></td>
<td><strong>7.8</strong></td>
</tr>
<tr class="even">
<td></td>
<td>OpenAI</td>
<td>39.6</td>
<td>52.1</td>
<td>53.4</td>
</tr>
<tr class="odd">
<td><a href="https://huggingface.co/KBLab/kb-whisper-small">small</a></td>
<td><strong>KBLab</strong></td>
<td><strong>7.3</strong></td>
<td><strong>6.4</strong></td>
<td><strong>6.6</strong></td>
</tr>
<tr class="even">
<td></td>
<td>OpenAI</td>
<td>20.6</td>
<td>26.4</td>
<td>26.4</td>
</tr>
<tr class="odd">
<td><a href="https://huggingface.co/KBLab/kb-whisper-medium">medium</a></td>
<td><strong>KBLab</strong></td>
<td><strong>6.6</strong></td>
<td><strong>5.4</strong></td>
<td><strong>5.8</strong></td>
</tr>
<tr class="even">
<td></td>
<td>OpenAI</td>
<td>12.1</td>
<td>15.8</td>
<td>17.1</td>
</tr>
<tr class="odd">
<td><a href="https://huggingface.co/KBLab/kb-whisper-large">large-v3</a></td>
<td><strong>KBLab</strong></td>
<td><strong>5.4</strong></td>
<td><strong>4.1</strong></td>
<td><strong>5.2</strong></td>
</tr>
<tr class="even">
<td></td>
<td>OpenAI</td>
<td>7.8</td>
<td>9.5</td>
<td>11.3</td>
</tr>
</tbody>
</table>
</div>
<p>Our evaluations show that the best-performing model reduces the WER by an average of 47% compared to Whisper-large-v3.</p>
<p>The results evaluated in terms of BLEU is presented in the table below.</p>
<div class="column-body">
<table class="caption-top table">
<thead>
<tr class="header">
<th>Model size</th>
<th></th>
<th>FLEURS</th>
<th>CommonVoice</th>
<th>NST</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>tiny</td>
<td>KBLab</td>
<td><strong>76.6</strong></td>
<td><strong>73.7</strong></td>
<td><strong>74.3</strong></td>
</tr>
<tr class="even">
<td></td>
<td>OpenAI</td>
<td>26.9</td>
<td>21.1</td>
<td>24.0</td>
</tr>
<tr class="odd">
<td>base</td>
<td>KBLab</td>
<td><strong>83.2</strong></td>
<td><strong>79.9</strong></td>
<td><strong>78.3</strong></td>
</tr>
<tr class="even">
<td></td>
<td>OpenAI</td>
<td>41.1</td>
<td>32.5</td>
<td>36.9</td>
</tr>
<tr class="odd">
<td>small</td>
<td>KBLab</td>
<td><strong>86.6</strong></td>
<td><strong>83.5</strong></td>
<td><strong>79.6</strong></td>
</tr>
<tr class="even">
<td></td>
<td>OpenAI</td>
<td>64.0</td>
<td>56.5</td>
<td>58.2</td>
</tr>
<tr class="odd">
<td>medium</td>
<td>KBLab</td>
<td><strong>87.6</strong></td>
<td><strong>85.0</strong></td>
<td><strong>80.2</strong></td>
</tr>
<tr class="even">
<td></td>
<td>OpenAI</td>
<td>77.1</td>
<td>70.1</td>
<td>68.9</td>
</tr>
<tr class="odd">
<td>large-v3</td>
<td>KBLab</td>
<td><strong>89.8</strong></td>
<td><strong>87.2</strong></td>
<td><strong>81.1</strong></td>
</tr>
<tr class="even">
<td></td>
<td>OpenAI</td>
<td>84.9</td>
<td>79.1</td>
<td>75.1</td>
</tr>
</tbody>
</table>
</div>
<p>The most significant improvements are observed in smaller models, demonstrating that high-quality transcriptions can be achieved with fewer computational resources.The KB-whisper-small model outperforms OpenAI’s whisper-large-v3, a model six times its size. ´This means that similar transcription quality can be obtained using a smaller model, making speech-to-text more accessible and less costly.</p>
</section>
<section id="where-to-find-the-models" class="level2">
<h2 class="anchored" data-anchor-id="where-to-find-the-models">Where to find the models?</h2>
<p>All models are freely available for download from KBLab’s page on <a href="https://huggingface.co/KBLab">HuggingFace</a>.</p>
<p>For more information on ASR models and how to use KBLab’s Whisper models programmatically, we recommend exploring this <a href="https://colab.research.google.com/drive/1RCP53jqClJz0zDX_VBT_K04BevUKk842?usp=sharing">notebook</a>.</p>
</section>
<section id="acknowledgments" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="acknowledgments">Acknowledgments</h2>
<p>We acknowledge the EuroHPC Joint Undertaking for awarding this project access to the EuroHPC supercomputer LEONARDO, hosted by CINECA (Italy) and the LEONARDO consortium, through the Development Access call and AI and data intensive applications access call.</p>

<div class="no-row-height column-margin column-container"><div class="">
<p><img src="https://kb-labb.github.io/posts/2025-03-07-welcome-KB-Whisper/images/eurohpc.png" class="img-fluid" style="margin-left: 5px;;width:35.0%"></p>
</div></div><p>The development work to produce the notebook mentioned above was carried out within the <a href="https://www.huminfra.se/">HUMINFRA</a> infrastructure project.</p>




<div class="no-row-height column-margin column-container"><div class="">
<p><img src="https://kb-labb.github.io/posts/2025-03-07-welcome-KB-Whisper/images/huminfra.svg" class="img-fluid" style="width:40.0%"></p>
</div></div></section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-bibliography"><h2 class="anchored quarto-appendix-heading">References</h2><div id="refs" class="references csl-bib-body hanging-indent" data-entry-spacing="0">
<div id="ref-wav2vec2" class="csl-entry">
Malmsten, Martin, Chris Haffenden, and Love Börjeson. 2022. <span>“Hearing Voices at the National Library – a Speech Corpus and Acoustic Model for the Swedish Language.”</span> <a href="https://arxiv.org/abs/2205.03026">https://arxiv.org/abs/2205.03026</a>.
</div>
<div id="ref-radford2022robustspeechrecognitionlargescale" class="csl-entry">
Radford, Alec, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. <span>“Robust Speech Recognition via Large-Scale Weak Supervision.”</span> <a href="https://arxiv.org/abs/2212.04356">https://arxiv.org/abs/2212.04356</a>.
</div>
</div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{vesterbacka2025,
  author = {Vesterbacka, Leonora and Rekathati, Faton and Kurtz, Robin
    and Sikora, Justyna and Toftgård, Agnes},
  title = {Welcome {KB-Whisper,} a New Fine-Tuned {Swedish} {Whisper}
    Model!},
  date = {2025-03-07},
  url = {https://kb-labb.github.io/posts/2025-03-07-welcome-KB-Whisper/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-vesterbacka2025" class="csl-entry quarto-appendix-citeas">
Vesterbacka, Leonora, Faton Rekathati, Robin Kurtz, Justyna Sikora, and
Agnes Toftgård. 2025. <span>“Welcome KB-Whisper, a New Fine-Tuned
Swedish Whisper Model!”</span> March 7, 2025. <a href="https://kb-labb.github.io/posts/2025-03-07-welcome-KB-Whisper/">https://kb-labb.github.io/posts/2025-03-07-welcome-KB-Whisper/</a>.
</div></div></section></div> ]]></description>
  <guid>https://kb-labb.github.io/posts/2025-03-07-welcome-KB-Whisper/</guid>
  <pubDate>Thu, 06 Mar 2025 23:00:00 GMT</pubDate>
  <media:content url="https://kb-labb.github.io/posts/2025-03-07-welcome-KB-Whisper/images/KBLab-whisper.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Preserving the history of cultural heritage conservation</title>
  <dc:creator>Chris Haffenden</dc:creator>
  <dc:creator>Emil Stenback</dc:creator>
  <link>https://kb-labb.github.io/posts/2025-01-31-preserving-the-history-of-cultural-heritage-conservation/</link>
  <description><![CDATA[ 





<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href="images/1.png" class="lightbox" data-gallery="quarto-lightbox-gallery-1" title="This page spread from Jonas Haquini Rhezelius, Runestones in Uppland (n.d. before 1666) - available in full here: Fc 8 - is one of the many sketch illustrations waiting to be found in KB’s F-collection."><img src="https://kb-labb.github.io/posts/2025-01-31-preserving-the-history-of-cultural-heritage-conservation/images/1.png" class="img-fluid figure-img" style="width:300.0%" alt="This page spread from Jonas Haquini Rhezelius, Runestones in Uppland (n.d. before 1666) - available in full here: Fc 8 - is one of the many sketch illustrations waiting to be found in KB’s F-collection."></a></p>
<figcaption>This page spread from Jonas Haquini Rhezelius, <em>Runestones in Uppland</em> (n.d. before 1666) - available in full here: <a href="https://www.manuscripta.se/ms/101823">Fc 8</a> - is one of the many sketch illustrations waiting to be found in KB’s F-collection.</figcaption>
</figure>
</div>
<section id="in-search-of-the-past" class="level2">
<h2 class="anchored" data-anchor-id="in-search-of-the-past">In search of the past</h2>
<p>How do we best preserve historical traces from oblivion? This question might feel particularly urgent today, as war, cyberattacks, and climate change create ever more acute threats to cultural heritage material <span class="citation" data-cites="bergvall2023 fredrikzon_haffenden_23">(Bergvall 2023; Frederikzon and Haffenden 2023)</span>. But the challenge is far from new. Historians like Peter Fritzsche have described how shifting perceptions of time after the French Revolution—and concerns about what was vanishing—contributed to the establishment of memory institutions across Europe in the mid-19th century <span class="citation" data-cites="Fritzsche2004sit Lowenthal1985tpi Swenson2013tro">(Fritzsche 2004; Lowenthal 1985; Swenson 2013)</span>. In Sweden, the story of heritage management goes even further back <span class="citation" data-cites="Jensen2018pse">(Jensen 2018)</span>. As early as the 17th century, organized efforts were underway to collect, document, and preserve physical cultural heritage, exemplified by Johannes Bureus (1568–1652), Sweden’s first antiquarian, national archivist, and royal librarian <span class="citation" data-cites="åström_2024">(Åström 2023)</span>.</p>
<p>Reading Bureus’ 400-year-old manuscripts about his journeys to document rune stones gives us a tangible sense of this particular history <span class="citation" data-cites="källström2024">(Källström 2024)</span>. His drawings, notebooks, and compilations of quotations remind those of us working with cultural heritage conservation today that we are part of a much longer tradition—one in which the similarities can often surprise, despite the obvious contrasts with our digital age. To paraphrase L. P. Hartley’s famous observation: the past may be a foreign country, but they didn’t always do things differently there.</p>
</section>
<section id="digitizing-the-history-of-cultural-heritage-conservation" class="level2">
<h2 class="anchored" data-anchor-id="digitizing-the-history-of-cultural-heritage-conservation">Digitizing the history of cultural heritage conservation</h2>
<p>Thanks to a recently started collaborative project between the National Library of Sweden (<em>Kungliga biblioteket</em>, KB) and the Swedish National Heritage Board (<em>Riksantikvarieämbetet</em>, RAÄ), it is now possible to explore Sweden’s early cultural heritage preservation efforts in an entirely new way. Funded by the Royal Swedish Academy of Letters, History, and Antiquities (<em>Vitterhetsakademien</em>), <a href="https://www.raa.se/2024/10/kulturminnesvardens-aldsta-historia-blir-tillganglig-for-alla/">the project is digitizing and making accessible a significant portion of this history</a>. For KB, this involves the entire F-collection of the library’s archival holdings, which includes Bureus’ manuscripts, while RAÄ is contributing a large number of archival documents from the 17th century to the late 19th century, drawn from its official archives.</p>
<p>Through digitization, this material—constituting the oldest history of RAÄ and the Academy of Letters—is being made available to researchers and the public alike. Via KB’s service <a href="https://www.manuscripta.se/search?q=rivih"><em>Manuscripta</em></a> and RAÄ’s <a href="https://app.raa.se/open/arkivsok/search"><em>Arkivsök</em></a>, researchers and the general public will be able to explore a newly reunited collection, previously scattered across several archives, that bears witness to the work of our predecessors in preserving the past.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href="images/2.jpg" class="lightbox" data-gallery="quarto-lightbox-gallery-2" title="Storage capsules, covers and boxes from many generations coexist in the F-collection. Some have withstood the test of time well, while others need to be inspected and replaced."><img src="https://kb-labb.github.io/posts/2025-01-31-preserving-the-history-of-cultural-heritage-conservation/images/2.jpg" class="img-fluid figure-img" style="width:300.0%" alt="Storage capsules, covers and boxes from many generations coexist in the F-collection. Some have withstood the test of time well, while others need to be inspected and replaced."></a></p>
<figcaption>Storage capsules, covers and boxes from many generations coexist in the F-collection. Some have withstood the test of time well, while others need to be inspected and replaced.</figcaption>
</figure>
</div>
</section>
<section id="preservation-measuresa-journey-of-discovery-in-itself" class="level2">
<h2 class="anchored" data-anchor-id="preservation-measuresa-journey-of-discovery-in-itself">Preservation measures—a journey of discovery in itself</h2>
<p>What does it mean in practice to digitize historical manuscripts and works? Image capture is obviously a central part of digitally preserving and making the material accessible, but significant work also needs to be carried out before the manuscripts even reach our photographers. The following post gives an insight into how we work at KB to conserve and prepare <a href="https://www.kb.se/hitta-och-bestall/om-samlingar-och-material/handskrifter.html">manuscript collections</a> for digitization.</p>
<p>Handling these materials in any way poses the risk of further damage, since the volumes and texts have been heavily used and are often extremely fragile. Many of them have lived rich and varied lives in the field before arriving in the library’s archives, and require careful and considered treatment. KB’s conservators and bookbinders therefore conduct thorough condition assessments and conservation measures before the material continues through the process. The close encounter with the material that this involves bears striking similarities to Bureus’ earlier explorations of ancient monuments and rune stones.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href="images/3.jpg" class="lightbox" data-gallery="quarto-lightbox-gallery-3" title="Reattaching a detached wax and paper seal."><img src="https://kb-labb.github.io/posts/2025-01-31-preserving-the-history-of-cultural-heritage-conservation/images/3.jpg" class="img-fluid figure-img" style="width:300.0%" alt="Reattaching a detached wax and paper seal."></a></p>
<figcaption>Reattaching a detached wax and paper seal.</figcaption>
</figure>
</div>
<p>First and foremost, each volume undergoes an evaluation before any further handling and image capture. Based on established criteria, the condition of the various components—bindings, sewing, text blocks, and media (e.g.&nbsp;ink)—is examined and documented. Different types of damage are graded according to their severity. For example, one volume might be in good condition with minor tears, while another might have extensive damage that risks worsening with handling. This evaluation determines whether conservation actions are needed and provides the best conditions for those handling the material during the rest of the digitisation process.</p>
<p>The assessment also considers whether the material is sensitive to changes in temperature and humidity, influencing where image capture can take place. For example, parchment covers tend to expand or shrink with climate fluctuations, and degraded ink risks cracking if pages “breathe” too much. Decisions are also made about whether new storage solutions are needed or if a damaged box must be replaced. In some cases, the conservators assist during digitization, such as in opening bindings at risk of breaking, loosening tight clasps, or carefully turning fragile pages.</p>
</section>
<section id="a-collection-full-of-surprises" class="level2">
<h2 class="anchored" data-anchor-id="a-collection-full-of-surprises">A collection full of surprises</h2>
<p>KB’s F-collection is a diverse and heterogeneous assemblage of bound handwritten documents, sketches of buildings, runes and landscapes, prints, and traditional texts annotated with marginalia. Every object in the collection has the potential to surprise, whether with striking imagery or something more unusual.</p>
<p>The first time we opened Martin Aschaneus’ <em>Collectaneum monetalium seu monetoscopia Sweogothica</em> (n.d. before 1641) - <a href="https://www.manuscripta.se/ms/101809">Fb 17:2</a> - we discovered coins bound between the pages on parchment strips. While books containing objects might seem more challenging to digitize, in this case, the binding method is ideal: the coins are as easy to leaf through as the pages! The thrill of discovering little treasures like this has been a recurring highlight throughout the project, and this coin book, one of the first volumes assessed, got us off to an exciting start.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href="images/4.jpg" class="lightbox" data-gallery="quarto-lightbox-gallery-4" title="Unexpected coins nesting as part of the book. Aschaneus’ Collectaneum monetalium seu monetoscopia Sweogothica (n.d. before 1641), Fb 17:2."><img src="https://kb-labb.github.io/posts/2025-01-31-preserving-the-history-of-cultural-heritage-conservation/images/4.jpg" class="img-fluid figure-img" style="width:300.0%" alt="Unexpected coins nesting as part of the book. Aschaneus’ Collectaneum monetalium seu monetoscopia Sweogothica (n.d. before 1641), Fb 17:2."></a></p>
<figcaption>Unexpected coins nesting as part of the book. Aschaneus’ <em>Collectaneum monetalium seu monetoscopia Sweogothica</em> (n.d. before 1641), <a href="https://www.manuscripta.se/ms/101809">Fb 17:2</a>.</figcaption>
</figure>
</div>
</section>
<section id="protecting-the-past-for-the-future" class="level2">
<h2 class="anchored" data-anchor-id="protecting-the-past-for-the-future">Protecting the past for the future</h2>
<p>We cannot turn back time, but with the right conservation measures and storage, we can extend the lifespan of materials at critical moments. Water damage mars the edges of many books. Ink spills have eaten through pages. Many bindings were originally temporary but have become permanent over time—easily mobile in the field but not particularly durable.</p>
<p>Even though digitization is a method of preserving collections, it is essential that the physical objects themselves, <em>the original</em>, remain in the best possible condition for future generations and research opportunities. With that in mind, tears and creases are among the most common damages that have needed to be repaired in the F-collection.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href="images/5.jpg" class="lightbox" data-gallery="quarto-lightbox-gallery-5" title="Shoring these fragments from ruin. Peringskiöld’s Om Heliga Birgitta (n.d.,1680-1720), Fh 26. Top right: before intervention. Bottom right: highlighting the damage of time. Left: after preservation measures."><img src="https://kb-labb.github.io/posts/2025-01-31-preserving-the-history-of-cultural-heritage-conservation/images/5.jpg" class="img-fluid figure-img" style="width:300.0%" alt="Shoring these fragments from ruin. Peringskiöld’s Om Heliga Birgitta (n.d.,1680-1720), Fh 26. Top right: before intervention. Bottom right: highlighting the damage of time. Left: after preservation measures."></a></p>
<figcaption>Shoring these fragments from ruin. Peringskiöld’s <em>Om Heliga Birgitta</em> (n.d.,1680-1720), Fh 26. Top right: before intervention. Bottom right: highlighting the damage of time. Left: after preservation measures.</figcaption>
</figure>
</div>
<p>A good example can be found in capsule Fh 26, Johan Peringskiöld’s <em>Om Heliga Birgitta</em> (n.d., between 1680 and 1720, <a href="https://libris.kb.se/bib/wc28f3h9t1xgdmvw">yet to be digitized</a>), which contains a letter where the same sheet of paper serves as both the writing surface and the envelope. At first glance, the letter appears somewhat worn, but as soon as we remove it from its capsule, we realize that it cannot be handled without falling apart (see red marking in figure 5). To make digitization possible, the folds need to be carefully smoothed out, and extremely thin Japanese paper is applied with adhesive to secure the different parts of the letter. This process is carried out with great care, and we avoid affecting the ink with the moisture introduced during treatment as much as possible.</p>
<p>Many of the volumes contain foldouts or plates that are larger than the rest of the pages and therefore had to be folded to fit within the covers. Johan Peringskiöld’s <em>Konungatal eller förteckning på Svea och Göta Rikes konungar och drottningar</em> (n.d. before 1720) - <a href="https://www.manuscripta.se/ms/101917">Fh 13</a> - is an example of a book that includes several large printed plates, one of which illustrates a procession and lists its participants. This particular plate is one of the largest, measuring a full 75 cm (see figure 6). Foldouts present several challenges both before and during digitization. The paper in the folds is often weakened, and long tears are common. These must be stabilized—not only to prevent the loss of material but also to ensure that text and images remain intact during digitization. It is also common for foldouts to have been improperly refolded over time, creating new creases that need to be smoothed out.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href="images/6.jpg" class="lightbox" data-gallery="quarto-lightbox-gallery-6" title="Mediating social order via extended diagrams. Peringskiöld, Konungatal eller förteckning på Svea och Göta Rikes konungar och drottningar, (n.d. before 1720), Fh 13."><img src="https://kb-labb.github.io/posts/2025-01-31-preserving-the-history-of-cultural-heritage-conservation/images/6.jpg" class="img-fluid figure-img" style="width:300.0%" alt="Mediating social order via extended diagrams. Peringskiöld, Konungatal eller förteckning på Svea och Göta Rikes konungar och drottningar, (n.d. before 1720), Fh 13."></a></p>
<figcaption>Mediating social order via extended diagrams. Peringskiöld, <em>Konungatal eller förteckning på Svea och Göta Rikes konungar och drottningar</em>, (n.d. before 1720), <a href="https://www.manuscripta.se/ms/101917">Fh 13</a>.</figcaption>
</figure>
</div>
</section>
<section id="only-the-beginning" class="level2">
<h2 class="anchored" data-anchor-id="only-the-beginning">Only the beginning…</h2>
<p>We are still in the early stages of the project. Some digitized items have already been published through <em>Manuscripta</em>, and at the time of writing, our conservators and bookbinders have assessed and treated 90 of approximately 300 volumes. Many challenges lie ahead, given the diversity of the collection. We are also exploring the possibility of applying the <a href="https://huggingface.co/blog/Gabriel/htrflow">Handwritten Text Recognition (HTR) models</a> developed by our colleagues at the Swedish National Archives’ <a href="https://huggingface.co/Riksarkivet">AI lab</a> to further enhance the material’s searchability and accessibility. We suspect more surprises await and hope that as you explore the digitized collection, you make your own discoveries—and perhaps gain a deeper appreciation for the dedication with which Johannes Bureus and so many others worked to preserve the heritage that continues to enrich our landscapes, both physical and digital.</p>
</section>
<section id="keen-to-explore-further" class="level2">
<h2 class="anchored" data-anchor-id="keen-to-explore-further">Keen to explore further?</h2>
<p>The material that has been digitized thus far in the project is freely available via <a href="https://www.manuscripta.se/search?q=rivih">Manuscripta</a>. Follow the link to browse the collection!</p>
<p>Read more about the collaborative project that enables this digitization work, RiViH, via the Swedish National Heritage Board’s <a href="https://www.raa.se/2024/10/kulturminnesvardens-aldsta-historia-blir-tillganglig-for-alla/">website</a> (in Swedish).</p>
<p>See how the research team attached to the project is making use of the material with this essay on the National Heritage Board’s <a href="https://k-blogg.se/tag/rivih-projektet/">K-blogg</a> (in Swedish).</p>



</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-bibliography"><h2 class="anchored quarto-appendix-heading">References</h2><div id="refs" class="references csl-bib-body hanging-indent" data-entry-spacing="0">
<div id="ref-åström_2024" class="csl-entry">
Åström, Patrik. 2023. <span>“Sammel­su­ri­et Sumlen.”</span> Samlingsbloggen, Kungliga biblioteket. <a href="https://www.kb.se/hitta-och-bestall/samlingsbloggen/blogginlagg/2024-08-26-sammelsuriet-sumlen.html">https://www.kb.se/hitta-och-bestall/samlingsbloggen/blogginlagg/2024-08-26-sammelsuriet-sumlen.html</a>.
</div>
<div id="ref-bergvall2023" class="csl-entry">
Bergvall, Greger. 2023. <span>“Hotat Kultur­arv Genom Tider­na.”</span> Samlingsbloggen, Kungliga biblioteket. <a href="https://www.kb.se/hitta-och-bestall/samlingsbloggen/blogginlagg/2024-09-05-hotat-kulturarv-genom-tiderna.html">https://www.kb.se/hitta-och-bestall/samlingsbloggen/blogginlagg/2024-09-05-hotat-kulturarv-genom-tiderna.html</a>.
</div>
<div id="ref-fredrikzon_haffenden_23" class="csl-entry">
Frederikzon, Johan, and Chris Haffenden. 2023. <span>“Towards Erasure Studies: Excavating the Material Conditions of Memory and Forgetting.”</span> <em>Memory, Mind &amp; Media</em> 2: 1–24. <a href="https://doi:10.1017/mem.2023.2">doi:10.1017/mem.2023.2</a>.
</div>
<div id="ref-Fritzsche2004sit" class="csl-entry">
Fritzsche, Peter. 2004. <em>Stranded in the Present : Modern Time and the Melancholy of History</em>. Cambridge Massachusetts: Harvard University Press.
</div>
<div id="ref-Jensen2018pse" class="csl-entry">
Jensen, Ola W. 2018. <em>P<span>å</span> Spaning Efter Det f<span>ö</span>rflutna : En Historia Om Arkeologiskt f<span>ä</span>ltarbete i Sverige 1600-1900</em>. Stockholm: Kungl. Vitterhets historie och antikvitets akademien (KVHAA).
</div>
<div id="ref-källström2024" class="csl-entry">
Källström, Magnus. 2024. <span>“Johannes Bureus Första Runresa.”</span> K-blogg,Riksantikvarieämbetet. <a href="https://k-blogg.se/2024/12/10/johannes-bureus-forsta-runresa/">https://k-blogg.se/2024/12/10/johannes-bureus-forsta-runresa/</a>.
</div>
<div id="ref-Lowenthal1985tpi" class="csl-entry">
Lowenthal, David. 1985. <em>The Past Is a Foreign Country</em>. Cambridge: Cambridge University Press.
</div>
<div id="ref-Swenson2013tro" class="csl-entry">
Swenson, Astrid. 2013. <em>The Rise of Heritage : Preserving the Past in France, Germany and England, 1789-1914</em>. Cambridge: Cambridge University Press.
</div>
</div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{haffenden2025,
  author = {Haffenden, Chris and Stenback, Emil},
  title = {Preserving the History of Cultural Heritage Conservation},
  date = {2025-01-31},
  url = {https://kb-labb.github.io/posts/2025-01-31-preserving-the-history-of-cultural-heritage-conservation/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-haffenden2025" class="csl-entry quarto-appendix-citeas">
Haffenden, Chris, and Emil Stenback. 2025. <span>“Preserving the History
of Cultural Heritage Conservation.”</span> January 31, 2025. <a href="https://kb-labb.github.io/posts/2025-01-31-preserving-the-history-of-cultural-heritage-conservation/">https://kb-labb.github.io/posts/2025-01-31-preserving-the-history-of-cultural-heritage-conservation/</a>.
</div></div></section></div> ]]></description>
  <guid>https://kb-labb.github.io/posts/2025-01-31-preserving-the-history-of-cultural-heritage-conservation/</guid>
  <pubDate>Thu, 30 Jan 2025 23:00:00 GMT</pubDate>
  <media:content url="https://kb-labb.github.io/posts/2025-01-31-preserving-the-history-of-cultural-heritage-conservation/images/1.png" medium="image" type="image/png" height="114" width="144"/>
</item>
<item>
  <title>Analysis of financial data at the Financial Supervisory Authority</title>
  <dc:creator>Jimmy Hollén</dc:creator>
  <link>https://kb-labb.github.io/posts/2024-01-30-finansinspektionen/</link>
  <description><![CDATA[ 





<p>Analyzing financial data from various market actors presents significant challenges. It’s not just about analyzing the data itself but also about handling the vast amount of data received regularly, both quantitatively and qualitatively. The data arrives on a spectrum from daily to yearly intervals.</p>
<p>Upon receiving the data, we meticulously analyze and categorize each institute into different risk categories. Subsequently, we devise a tailored supervisory plan for each institute, with the most high-risk institutions receiving the most comprehensive action plans.</p>
<p>Quantitative data undergoes processing and analysis through relevant software, while qualitative data, typically in PDF format, is scrutinized mostly by supervisors. Analyzing qualitative data is resource-intensive and prone to oversights.</p>
<p>To streamline the supervision of qualitative data, we’ve developed a machine learning model with support from the European Commission, DG Reform, and the Research Institute in Sweden (RISE). During the model’s development, we utilized KBLab’s Named Entity Recognition model based on KB-BERT to minimize noise. More precisely KBLab’s model helped us anonymize the data to minimize spurious correlations. The model has proven effective, prompting us to further enhance its capabilities. So instead of roughly 20 people reading little less than 200 reports we nowadays spend roughly one hour for the same work.</p>
<p>As we progress, our commitment to refining our analytical capabilities remains steadfast. By leveraging cutting-edge technologies and collaborative partnerships, we strive to enhance the efficacy of financial market supervision and regulation.</p>



<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{hollén2024,
  author = {Hollén, Jimmy},
  title = {Analysis of Financial Data at the {Financial} {Supervisory}
    {Authority}},
  date = {2024-01-30},
  url = {https://kb-labb.github.io/posts/2024-01-30-finansinspektionen/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-hollén2024" class="csl-entry quarto-appendix-citeas">
Hollén, Jimmy. 2024. <span>“Analysis of Financial Data at the Financial
Supervisory Authority.”</span> January 30, 2024. <a href="https://kb-labb.github.io/posts/2024-01-30-finansinspektionen/">https://kb-labb.github.io/posts/2024-01-30-finansinspektionen/</a>.
</div></div></section></div> ]]></description>
  <guid>https://kb-labb.github.io/posts/2024-01-30-finansinspektionen/</guid>
  <pubDate>Mon, 29 Jan 2024 23:00:00 GMT</pubDate>
  <media:content url="https://kb-labb.github.io/posts/2024-01-30-finansinspektionen/finansinspektionen.JPG" medium="image"/>
</item>
<item>
  <title>Unearthing forgotten images with the help of AI</title>
  <dc:creator>Chris Haffenden</dc:creator>
  <dc:creator>Faton Rekathati</dc:creator>
  <dc:creator>Emma Rende</dc:creator>
  <link>https://kb-labb.github.io/posts/2023-10-20-unearthing-forgotten-images-with-the-help-of-ai/</link>
  <description><![CDATA[ 





<section id="forgotten-in-the-archive" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="forgotten-in-the-archive">Forgotten in the archive…</h2>
<p>Memory institutions like KB can often be faced with a palpable sense of overload. Partly this is an effect of the proliferation of new material now being produced by digital culture. With the number of items delivered to the library via electronic legal deposit rapidly increasing, how is any sort of order to be maintained? But it is also a question of dealing with inherited archival blindspots, where previous historical moments of mass media expansion created the conditions for parts of the collections to remain undescribed. Without the metadata that would make them discoverable, such items have now largely been consigned to the realm of forgetting that Aleida Assmann has described as “the passively stored memory that preserves the past” <span class="citation" data-cites="Assmann_2010">(Assmann 2010)</span>.</p>
<aside>
The image search demo can be accessed at <a href="https://lab.kb.se/bildsok">https://lab.kb.se/bildsok</a>
</aside>
<p>A pertinent example of a material that, while preserved, lacks description - perhaps due to its perceived ephemerality historically and the limited valuation this has granted in terms of archival resources and attention - is visual heritage collections from the nineteenth and twentieth century. Descriptive cataloguing has long been a central part of KB’s making its collections accessible and searchable, but it has been impossible for each and every incoming item to be manually catalogued, given that legal deposit legislation dictates the library receives a copy of <em>everything</em> published in Sweden. Instead, certain items such as postcards or adverts have tended to be grouped together and classified under collective catalogue entries that often preclude the detailing of any specific information about the individual object per se.</p>
<p>Although KB has a rich and diverse collection of <a href="https://arken.kb.se/se-s-kob-vykort">c.&nbsp;600,000 postcards</a>, the lack of navigability and overview entrenched by scarce metadata has made the material hard for users to access - or even to be aware of its existence. Despite being preserved as part of our shared cultural heritage, such items are thus at risk of disappearing from view altogether.</p>
</section>
<section id="multimodal-ai-as-novel-entrance-point" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="multimodal-ai-as-novel-entrance-point">Multimodal AI as novel entrance point</h2>
<p>Recent developments within AI offer a promising strategy for libraries and other GLAM institutions with large collections of undescribed visual heritage to counter this archival forgetting and make such material more visible. The emergence of powerful multimodal AI models makes it possible to interact with huge amounts of images in new ways, with or without preexisting metadata. This is evident, for example, in the computer vision technology underpinning Google’s image search function, which we now tend to take for granted on the phones in our pockets.</p>
<aside>
<strong>GLAM:</strong> <strong>G</strong>alleries, <strong>L</strong>ibraries, <strong>A</strong>rchives and <strong>M</strong>useums.
</aside>
<p>By connecting the visual and textual domains in an innovative manner, multimodal AI has enabled more effective online search techniques, both in terms of text-to-image (i.e.&nbsp;image retrieval from a textual description) and image-to-image search (otherwise known as reverse image search, i.e.&nbsp;image retrieval from an image). The Open AI model <a href="https://openai.com/research/clip">CLIP</a> was trained on a dataset of 400 million matching text-image pairs to learn to recognize visual concepts and their associated names - e.g.&nbsp;an image of a cat and “a photo of a cat” <span class="citation" data-cites="radford2021learning">(Radford et al. 2021)</span>. This works via the use of vector space: through transforming the input text or image to a numerical representation (called <a href="https://en.wikipedia.org/wiki/Word_embedding">embeddings</a>), the AI model can return results based upon images with a similar representation (see fig.&nbsp;@ref(fig:embeddings)). When “church” appears in a text, for instance, the model will generate a similar number representation for both the word description and the visual manifestation of a church, thus collapsing the distinction between the different modes and making them directly comparable. Such a technique can be used to identify all the images containing a church in a large collection of images, regardless of whether these images have previously been described as such. In short, it offers content-based image search independent of prior metadata.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href="images/embeddings.png" class="lightbox" data-gallery="quarto-lightbox-gallery-1" title="Matching similar images and texts within vector space. Image: Federico Bianchi."><img src="https://kb-labb.github.io/posts/2023-10-20-unearthing-forgotten-images-with-the-help-of-ai/images/embeddings.png" class="img-fluid figure-img" style="width:100.0%" alt="Matching similar images and texts within vector space. Image: Federico Bianchi."></a></p>
<figcaption>Matching similar images and texts within vector space. Image: Federico Bianchi.</figcaption>
</figure>
</div>
</section>
<section id="applying-multimodal-search-at-the-library" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="applying-multimodal-search-at-the-library">Applying multimodal search at the library</h2>
<p>We chose to explore the potential of this method for heritage material through a pilot project focused upon a selection of KB’s postcards. While the original version of CLIP was trained upon English text data, we turned to the Swedish adaptation of the model produced by <span class="citation" data-cites="carlsson-etal-2022-cross">(Carlsson et al. 2022)</span>, Swe-CLIP 2M, to enable free text search in Swedish. Since only part of the library’s postcard collection has been digitized, we could include 17,409 postcards in the project. Given that the back side of each postcard could include personal details such as names and addresses, we opted to employ only the front sides. Using this digital material as a dataset, we sought to investigate the relevance of cutting-edge AI search techniques in a library context.</p>
<div class="page-columns page-full">
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p class="page-columns page-full"><a href="images/interface.png" class="lightbox page-columns page-full" data-gallery="quarto-lightbox-gallery-2" title="Interface for postcard image search."><img src="https://kb-labb.github.io/posts/2023-10-20-unearthing-forgotten-images-with-the-help-of-ai/images/interface.png" class="img-fluid figure-img column-body-outset" style="width:100.0%" alt="Interface for postcard image search."></a></p>
<figcaption>Interface for postcard image search.</figcaption>
</figure>
</div>
</div>
<p>After close collaboration with the library’s developers, the project resulted in an image search demo, which, thanks to a recent licensing agreement, can now be openly accessed <a href="https://lab.kb.se/bildsok">here</a>. This amounts to an interface that allows these postcards to be searched according to either text or image search terms (see fig.&nbsp;@ref(fig:interface)), making a previously largely undescribed - and therefore undiscoverable - material amenable to the sort of granular search possibilities we are familiar with from online search engines.</p>
<p>The demo works by exploiting the multimodal capabilities of CLIP outlined above. So any text or image entered into the search box is run through the model, transformed into a numerical representation, and then compared with the equivalent representations for all of the postcards that have been previously processed and stored in a database. The search results are ranked according to (cosine) similarity, with the postcards with the closest representation to the search term appearing at the top of the list. Insofar as it prototypes a new entrance point to the collections, our demo provides a striking example of the transformative possibilities of vector databases for memory institutions.</p>
</section>
<section id="test-and-explore" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="test-and-explore">Test and explore!</h2>
<p>As part of our wider mission at KBLab to contribute to wider use and discussion of AI tools for the heritage sector and beyond <span class="citation" data-cites="börjeson_haffenden_malmsten_klingwall_rende_kurtz_rekathati_hägglöf_sikora_2023">(Börjeson et al. 2023)</span>, we encourage you to try out the image search demo and play around with its capabilities. Here we would like to offer a few brief pointers about how to use the demo and understand the results.</p>
<p>The first option is free text search: you can provide the search box with either a general term - i.e.&nbsp;“kyrka” (fig.&nbsp;@ref(fig:kyrka)) - or a more specific set of terms, if you would like to find something more particular - i.e.&nbsp;“kyrka och blå himmel bilar” (fig.&nbsp;@ref(fig:kyrkabilar)). The second option is image search. Here you can either click on a given image in order to be provided with a results list of the most similar images in the collection (fig.&nbsp;@ref(fig:imagetoimage)), or you can upload your own image to find out which of the postcards are most closely related. (This latter option is particularly useful if you are a researcher interested in tracing the reception history and transmission of a particular image.) As the green tags at the top of figure @ref(fig:imagetoimage) suggest, the images have also been selectively enriched with various tags to enhance searchability: you can click on a tag to see which postcards share the same description, i.e.&nbsp;the category “Borgholms kyrka”.</p>
<div class="page-columns page-full">
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p class="page-columns page-full"><a href="images/kyrka.png" class="lightbox page-columns page-full" data-gallery="quarto-lightbox-gallery-3" title="Free text search in the postcard collection for the Swedish term “church”."><img src="https://kb-labb.github.io/posts/2023-10-20-unearthing-forgotten-images-with-the-help-of-ai/images/kyrka.png" class="img-fluid figure-img column-body-outset" style="width:100.0%" alt="Free text search in the postcard collection for the Swedish term “church”."></a></p>
<figcaption>Free text search in the postcard collection for the Swedish term “church”.</figcaption>
</figure>
</div>
</div>
<div class="page-columns page-full">
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p class="page-columns page-full"><a href="images/kyrka_bilar.png" class="lightbox page-columns page-full" data-gallery="quarto-lightbox-gallery-4" title="Search results for the Swedish search string: “church and blue sky cars”."><img src="https://kb-labb.github.io/posts/2023-10-20-unearthing-forgotten-images-with-the-help-of-ai/images/kyrka_bilar.png" class="img-fluid figure-img column-body-outset" style="width:100.0%" alt="Search results for the Swedish search string: “church and blue sky cars”."></a></p>
<figcaption>Search results for the Swedish search string: “church and blue sky cars”.</figcaption>
</figure>
</div>
</div>
<div class="page-columns page-full">
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p class="page-columns page-full"><a href="images/image_to_image.png" class="lightbox page-columns page-full" data-gallery="quarto-lightbox-gallery-5" title="Image-to-image search to find most closely related postcards."><img src="https://kb-labb.github.io/posts/2023-10-20-unearthing-forgotten-images-with-the-help-of-ai/images/image_to_image.png" class="img-fluid figure-img column-body-outset" style="width:100.0%" alt="Image-to-image search to find most closely related postcards."></a></p>
<figcaption>Image-to-image search to find most closely related postcards.</figcaption>
</figure>
</div>
</div>
<p>When it comes to understanding the search results, an important feature that needs to be considered is the production of relative similarity. This means that the model will always return a result of the 100 “closest results”, even if there are no exact matches for the given search term in the database. If we search for the Swedish term “cat”, for example, the top results include postcards of a lynx, a goat and a bear, but no cats (fig.&nbsp;@ref(fig:katt)). This is not because the model has misunderstood the search term, but rather that the particular data in the database happens not to include any such items, and that these other animals were the closest visual concepts identifiable in the data.</p>
<div class="page-columns page-full">
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p class="page-columns page-full"><a href="images/katt.png" class="lightbox page-columns page-full" data-gallery="quarto-lightbox-gallery-6" title="Searching for the non-existent “cat” in the postcard collection."><img src="https://kb-labb.github.io/posts/2023-10-20-unearthing-forgotten-images-with-the-help-of-ai/images/katt.png" class="img-fluid figure-img column-body-outset" style="width:100.0%" alt="Searching for the non-existent “cat” in the postcard collection."></a></p>
<figcaption>Searching for the non-existent “cat” in the postcard collection.</figcaption>
</figure>
</div>
</div>
</section>
<section id="a-glimpse-of-the-future" class="level2">
<h2 class="anchored" data-anchor-id="a-glimpse-of-the-future">A glimpse of the future?</h2>
<p>With this postcard demo project, we have illustrated the capacity for multimodal AI and vector databases to improve the searchability and accessibility of cultural heritage collections. As our specific case has shown, these techniques are particularly relevant as a means of transforming access to heritage material lacking metadata, with the multimodal dimension especially pertinent for visual heritage.</p>
<p>There are certain practical preconditions for the adoption of such technology in a heritage setting, most of which can be related to questions of funding: that the material has been digitized, that there are sufficient resources available for computation, data science and developer expertise, and that there are licensing agreements in place. But in broad brushstrokes, our pilot project still suggests a captivating vision of what the future of an AI-underpinned memory institution might look like with a little help from machine learning.</p>
</section>
<section id="acknowledgements" class="level2">
<h2 class="anchored" data-anchor-id="acknowledgements">Acknowledgements</h2>
<p>A major part of the development and design of the postcard demo was carred out by KB’s developers, without whose efforts it would look nowhere near as pretty and be nowhere near as efficient. We would like to thank Matthias Nilsson, Krzysztof Bergendahl and Ebrima Faye for their work on the project!</p>



</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-bibliography"><h2 class="anchored quarto-appendix-heading">References</h2><div id="refs" class="references csl-bib-body hanging-indent" data-entry-spacing="0">
<div id="ref-Assmann_2010" class="csl-entry">
Assmann, Aleida. 2010. <span>“Canon and Archive.”</span> In <em>Cultural Memory Studies: An International and Interdisciplinary Handbook</em>, edited by Astrid Erll and AnsgarEditors Nünning, 97–108. De Gruyter.
</div>
<div id="ref-börjeson_haffenden_malmsten_klingwall_rende_kurtz_rekathati_hägglöf_sikora_2023" class="csl-entry">
Börjeson, Love, Chris Haffenden, Martin Malmsten, Fredrik Klingwall, Emma Rende, Robin Kurtz, Faton Rekathati, Hillevi Hägglöf, and Justyna Sikora. 2023. <span>“Transfiguring the Library as Digital Research Infrastructure: Making KBLab at the National Library of Sweden.”</span> SocArXiv. <a href="https://doi.org/10.31235/osf.io/w48rf">https://doi.org/10.31235/osf.io/w48rf</a>.
</div>
<div id="ref-carlsson-etal-2022-cross" class="csl-entry">
Carlsson, Fredrik, Philipp Eisen, Faton Rekathati, and Magnus Sahlgren. 2022. <span>“Cross-Lingual and Multilingual <span>CLIP</span>.”</span> In <em>Proceedings of the Thirteenth Language Resources and Evaluation Conference</em>, 6848–54. Marseille, France: European Language Resources Association. <a href="https://aclanthology.org/2022.lrec-1.739">https://aclanthology.org/2022.lrec-1.739</a>.
</div>
<div id="ref-radford2021learning" class="csl-entry">
Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, et al. 2021. <span>“Learning Transferable Visual Models from Natural Language Supervision.”</span> <a href="https://arxiv.org/abs/2103.00020">https://arxiv.org/abs/2103.00020</a>.
</div>
</div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{haffenden2023,
  author = {Haffenden, Chris and Rekathati, Faton and Rende, Emma},
  title = {Unearthing Forgotten Images with the Help of {AI}},
  date = {2023-10-20},
  url = {https://kb-labb.github.io/posts/2023-10-20-unearthing-forgotten-images-with-the-help-of-ai/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-haffenden2023" class="csl-entry quarto-appendix-citeas">
Haffenden, Chris, Faton Rekathati, and Emma Rende. 2023.
<span>“Unearthing Forgotten Images with the Help of AI.”</span> October
20, 2023. <a href="https://kb-labb.github.io/posts/2023-10-20-unearthing-forgotten-images-with-the-help-of-ai/">https://kb-labb.github.io/posts/2023-10-20-unearthing-forgotten-images-with-the-help-of-ai/</a>.
</div></div></section></div> ]]></description>
  <guid>https://kb-labb.github.io/posts/2023-10-20-unearthing-forgotten-images-with-the-help-of-ai/</guid>
  <pubDate>Thu, 19 Oct 2023 22:00:00 GMT</pubDate>
  <media:content url="https://kb-labb.github.io/posts/2023-10-20-unearthing-forgotten-images-with-the-help-of-ai/images/bildsok.png" medium="image" type="image/png" height="76" width="144"/>
</item>
<item>
  <title>Words unboxed: discovering new words with Kubord</title>
  <dc:creator>Markus Forsberg</dc:creator>
  <dc:creator>Justyna Sikora</dc:creator>
  <dc:creator>Emma Sköldberg</dc:creator>
  <link>https://kb-labb.github.io/posts/2023-08-29-kubord/</link>
  <description><![CDATA[ 





<p>Since the last century, linguists have gradually become increasingly interested in quantitative research. One of the most well-known examples of how linguistics can benefit from quantitative methods is the story of the American linguist George Kingsley Zipf. In the 1930s, Zipf broadened his scientific focus from linguistics to statistics. This fascination led to the formulation of two laws: the law of abbreviation and a law addressing frequency distribution. The latter, known simply as Zipf’s law, states that the distribution of words in a natural language is inversely proportional to their rank in the frequency distribution. For example, the second most common word in a corpus occurs only half as frequently as the first one, the third word is half as frequent as the second one, and so on.</p>
<p>This law has been used in various research projects, including the analysis of the Voynich manuscript and, more recently, the investigation of AI-generated texts.</p>
<p>Advancements in quantitative research such as language technology have also proven to be extremely helpful in other fields of language research such as lexicography. Language researchers have, for a long time ago, discovered the potential of exploring language statistics, as they provide us with important information about language use, such as word formation and language change. In this regard, newspaper data seem to be an excellent source for identifying the emergence of new words or tracking changes in the meaning of existing ones, since this data typically reflect the language used at the time it was published. Books, on the other hand, while providing vast amounts of high-quality data, the lengthy publishing process and the specific nature of literary language make them less anchored in time, hence not as fitting for the aforementioned purposes compared to newspapers.</p>
<section id="kubord-and-kubord-2" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="kubord-and-kubord-2">Kubord and Kubord 2</h3>
<p>With this in mind, <a href="https://kb-labb.github.io/">KBLab</a> together with <a href="https://spraakbanken.gu.se/">Språkbanken Text</a>, has been working towards the development of freely available datasets to support research in, but not limited to, lexicography. We have aimed at maximizing the usefulness of the data for researchers while at the same time respecting our copyright protection laws, and the result of these efforts are Kubord – a data set containing annotated word frequencies from modern Swedish newspapers. To conduct the analysis, the texts have been processed using the text analysis tool <a href="https://spraakbanken.gu.se/sparv">Sparv</a>, developed at Språkbanken Text. In addition to calculating word statistics, the newspaper data has been automatically annotated with additional linguistic information such as part-of-speech tags and lemmas.</p>
<p>The original Kubord data set consists of 84 datasets from Dagens Nyheter, Svenska Dagbladet, Aftonbladet, Expressen, Östgöta Correpondenten, and Göteborgsposten. The majority of the data spans from 2010 to 2021; however, word frequencies for Dagens Nyheter from 2001-2009 are also available.</p>
<p>Extending the work on Kubord, 75 new datasets, marked as Kubord 2, have recently been developed and made publicly available. Kubord 2 can be seen as an enriched version of the original corpora. The novelty lies in that the analyses are supplemented with information about the frequency of the pairs of syntactically related words, such as a word acting as the subject paired with a word acting as the main verb, for all the words present in the data. Similar to Kubord, Kubord 2 covers the years 2010-2021, and the aim is to update the collection with newer data as it becomes available.</p>

<div class="no-row-height column-margin column-container"><div class="">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://kb-labb.github.io/posts/2023-08-29-kubord/kubord2.png" class="img-fluid figure-img"></p>
<figcaption>Kubord 2 data available in the research tool Korp.</figcaption>
</figure>
</div>
</div></div></section>
<section id="making-the-most-of-kubord" class="level3">
<h3 class="anchored" data-anchor-id="making-the-most-of-kubord">Making the most of Kubord</h3>
<p>The usefulness of having Kubord available for lexicographers has been described with examples in the Språkbanken blog post titled <a href="https://spraakbanken.gu.se/blogg/index.php/2022/11/28/hur-fangar-vi-upp-svenskans-nya-ord-med-hjalp-av-kubord/">“Hur fångar vi upp svenskans nya ord med hjälp av Kubord?”</a> (How do we capture new Swedish words with the help of Kubord?).</p>
<p>As highlighted in the article, tracking changes in word popularity (measured by frequency counts) can provide insight into changes in society - how our eating habits evolve, how we spend our free time, and more. It can therefore serve as a valuable source of knowledge for humanistic researchers such as linguists, historians, or sociologists. Additionally, it sheds light on which words come into use and when, which is especially useful in the context of lexicographical projects such as the Swedish Academy contemporary dictionary project (Svenska Akademiens samtidsordböcker).</p>
<p>Thanks to the annotated word statistics obtained from Kubord, we can conclude that some words that became more prominent during 2020-2021 were:</p>
<p><em>barnfridsbrott, charkbricka, freeskiing, fucking, gastropub, glamping, hjärndimma, höghöjdsbana, incel, intimitetskoordinator, jätteloka, kakuro, klassikerskydd, kulturtolk, lockdown, magnetfiskare, mikrostat, minoritetsstress, miso, mockumentär, mobildata, mobilitetshus, nagelsalong, naturhus, nettonollutsläpp, näringsterapeut, parallellsamhälle, powerwalk, preklinisk, prosecco, reduktionsplikt, rättspsyk, röstassistent, salsiccia, sexsomni, skidalpinism, skills, skyddsperson, sportswashing, streetfood, syrah, trygghetspension, uppläggningskostnad, utsläppssiffra, verkställighetshinder, villkorstrappa, yes</em>.</p>
<p>The majority of these words will most likely be incorporated in future editions of the Swedish Academy’s contemporary dictionaries.</p>
</section>
<section id="using-the-research-tool-korp-for-kubord" class="level3">
<h3 class="anchored" data-anchor-id="using-the-research-tool-korp-for-kubord">Using the research tool Korp for Kubord</h3>
<p><a href="https://spraakbanken.gu.se/korp">Korp</a>, a research tool developed at Språkbanken Text, is a tool that supports quantitative research of (sequences of) annotated words, and is heavily used by the Swedish lexicographers. Since Kubord data set is restricted to words rather than texts, not all of Korp’s functionalities are available, but some of the most important ones are, such as the statistics and the word picture mode. As an example, when a lexicographer investigates different kinds of word combinations, a word picture is useful, as it gives an overview of selected syntactical environments of a word. For nouns the word picture contains typical verbs, prepositions, pre-modifier, and post-modifiers, etc.</p>
<p>Below you find <a href="https://spraakbanken.gu.se/korp/?mode=kubord#?cqp=%5B%5D&amp;corpus=kubord2-dn-2010,kubord2-dn-2011,kubord2-dn-2012,kubord2-dn-2013,kubord2-dn-2014,kubord2-dn-2015,kubord2-dn-2016,kubord2-dn-2017,kubord2-dn-2018,kubord2-dn-2019,kubord2-dn-2020,kubord2-dn-2021,kubord2-svd-2010,kubord2-svd-2011,kubord2-svd-2012,kubord2-svd-2013,kubord2-svd-2014,kubord2-svd-2015,kubord2-svd-2016,kubord2-svd-2017,kubord2-svd-2018,kubord2-svd-2019,kubord2-svd-2020,kubord2-svd-2021&amp;word_pic&amp;search=lemgram%7Cavtal%5C.%5C.nn%5C.1&amp;page=0&amp;result_tab=3">the word picture of the Swedish noun avtal</a>, based on Kubord 2.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://kb-labb.github.io/posts/2023-08-29-kubord/wordpicture.png" class="img-fluid figure-img" style="width:100.0%"></p>
<figcaption>Word picture of the Swedish noun avtal</figcaption>
</figure>
</div>
<p>Some of the words in the first column of the word picture represent more or less free combinations that include <em>avtal</em>, e.g.&nbsp;<em>enligt (ett) avtal och med stöd av (ett) avtal</em>. In the column with pre-modifiers you find <em>(ett) nytt/skriftligt/muntligt/bilateralt/internationellt avtal</em>, and more. This information is useful for a lexicographer when looking for good and typical language examples of the current word, which must support the meaning descriptions and, at the same time, demonstrate how the word is typically used.</p>
<p>Some of the words and phrases that appear in the word picture also support a lexicographer’s work on providing valency information concerning <em>avtal</em> (see e.g.&nbsp;the post-modifier including <em>med</em>).</p>
<p>Finally, the word picture provides a lexicographer with recurrent word combinations including the noun and a verb, e.g.&nbsp;<em>teckna/sluta/ingå/underteckna/löpa ut/säga upp ett avtal</em> (see the columns 4–5).</p>
<p>Now, let’s explore how we can supplement this example with information from the Kubord 2 data set by examining some of the new Swedish words mentioned above:</p>
<ul>
<li><a href="https://spraakbanken.gu.se/korp/?mode=kubord#?cqp=%5B%5D&amp;corpus=kubord2-dn-2010,kubord2-dn-2011,kubord2-dn-2012,kubord2-dn-2013,kubord2-dn-2014,kubord2-dn-2015,kubord2-dn-2016,kubord2-dn-2017,kubord2-dn-2018,kubord2-dn-2019,kubord2-dn-2020,kubord2-dn-2021,kubord2-svd-2010,kubord2-svd-2011,kubord2-svd-2012,kubord2-svd-2013,kubord2-svd-2014,kubord2-svd-2015,kubord2-svd-2016,kubord2-svd-2017,kubord2-svd-2018,kubord2-svd-2019,kubord2-svd-2020,kubord2-svd-2021&amp;word_pic&amp;result_tab=3&amp;search=word%7Cstreetfood&amp;page=0">streetfood</a></li>
</ul>
<p>The word picture of <em>streetfood</em> illustrates that this particular noun is often preceded by adjectives such as <em>asiatisk</em>, <em>mexikansk</em>, <em>kantonesisk</em> and <em>vegansk</em>. These adjectives clearly show the large selection, the international elements and the variety when it comes to street food of today. Furthermore, in written texts <em>streetfood</em> often is preceded by verbs such as <em>servera</em>, <em>laga</em>, <em>sälja</em>, <em>äta</em> och <em>avnjuta</em>. Obviously, both the seller’s and the buyer’s perspective on the sale of this type of food are represented in the list.</p>
<ul>
<li><a href="https://spraakbanken.gu.se/korp/?mode=kubord#?cqp=%5B%5D&amp;corpus=kubord2-dn-2010,kubord2-dn-2011,kubord2-dn-2012,kubord2-dn-2013,kubord2-dn-2014,kubord2-dn-2015,kubord2-dn-2016,kubord2-dn-2017,kubord2-dn-2018,kubord2-dn-2019,kubord2-dn-2020,kubord2-dn-2021,kubord2-svd-2010,kubord2-svd-2011,kubord2-svd-2012,kubord2-svd-2013,kubord2-svd-2014,kubord2-svd-2015,kubord2-svd-2016,kubord2-svd-2017,kubord2-svd-2018,kubord2-svd-2019,kubord2-svd-2020,kubord2-svd-2021&amp;word_pic&amp;result_tab=3&amp;page=0&amp;search=word%7Clockdown">lockdown</a></li>
</ul>
<p>A large number of more or less new words became more established in Swedish during the pandemic. One of those words is <em>lockdown</em>. (The more Swedish-sounding <em>nedstängning</em> was also established but was not much used.) When it comes to lexicographic work, it can be discussed which of these pandemic related words are temporary and which are here to stay. A search in all texts in KB’s Svenska dagstidningar shows that the use of the word <a href="https://tidningar.kb.se/?q=lockdown"><em>lockdown</em></a> has decreased dramatically this year.</p>
<p>A word picture for <em>lockdown</em> based on Kubord 2 gives a richer view of the word and shows that the actual shutdown is often specified by attributes such as <em>total</em>, <em>(sten)hård</em>, <em>strikt</em>, <em>fullskalig</em> och <em>fullständig</em>, but that a lockdown can also be <em>partiell</em>, and in newspaper texts, the word is often a subject to the verb <em>införa</em> and an object to the verb combination <em>träda i kraft</em>. All this relevant information if the word ever finds its way into one of the Swedish dictionaries.</p>
<ul>
<li><a href="https://spraakbanken.gu.se/korp/?mode=kubord#?cqp=%5B%5D&amp;corpus=kubord2-dn-2010,kubord2-dn-2011,kubord2-dn-2012,kubord2-dn-2013,kubord2-dn-2014,kubord2-dn-2015,kubord2-dn-2016,kubord2-dn-2017,kubord2-dn-2018,kubord2-dn-2019,kubord2-dn-2020,kubord2-dn-2021,kubord2-svd-2010,kubord2-svd-2011,kubord2-svd-2012,kubord2-svd-2013,kubord2-svd-2014,kubord2-svd-2015,kubord2-svd-2016,kubord2-svd-2017,kubord2-svd-2018,kubord2-svd-2019,kubord2-svd-2020,kubord2-svd-2021&amp;word_pic&amp;prefix&amp;page=0&amp;result_tab=3&amp;show_stats&amp;search=word%7Cprekliniska">preklinisk</a></li>
</ul>
<p>Among the words that are more frequently used in recent years you also find the adjective <em>preklinisk</em>. A search in KB’s Svenska dagstidningar shows that the word has been used since at least 1945 in Swedish texts. The adjective also has different meanings in modern Swedish. For example, when <em>preklinisk</em> is followed by words like <em>läkemedelsutveckling</em>, <em>studie(r)</em> och <em>test(er)</em> in its word picture, it is with the meaning ‘relating to drug testing’. When <em>preklinisk</em>, on the other hand, is used together with the noun <em>fas(er)</em>, it is with the meaning ‘which applies to the early stage of a certain disease’. In other words, the word picture view gives support in sense differentiation.</p>
<p>A Korp search also shows that it is, above all, the word form <em>prekliniska</em> (as in <em>prekliniska tester</em>) that is used in Kubord 2 and it is also something that a lexicographer takes into account when giving typical examples of how the word is used.</p>
<ul>
<li><a href="https://spraakbanken.gu.se/korp/?mode=kubord#?cqp=%5B%5D&amp;corpus=kubord2-dn-2010,kubord2-dn-2011,kubord2-dn-2012,kubord2-dn-2013,kubord2-dn-2014,kubord2-dn-2015,kubord2-dn-2016,kubord2-dn-2017,kubord2-dn-2018,kubord2-dn-2019,kubord2-dn-2020,kubord2-dn-2021,kubord2-svd-2010,kubord2-svd-2011,kubord2-svd-2012,kubord2-svd-2013,kubord2-svd-2014,kubord2-svd-2015,kubord2-svd-2016,kubord2-svd-2017,kubord2-svd-2018,kubord2-svd-2019,kubord2-svd-2020,kubord2-svd-2021&amp;word_pic&amp;prefix&amp;page=0&amp;result_tab=3&amp;show_stats&amp;search=word%7Cverkställighetshinder">verkställighetshinder</a></li>
</ul>
<p>A final example word is <em>verkställighetshinder</em>. It has increased strongly in use in Swedish texts since around 2012. The word is primarily used in the context of migration, and this context is clearly evident as that the name Migrationsverk is present in the word picture. Its word picture further reveals that the noun is used in, at least, two different contexts. On the one hand, a person can <em>anmäla</em> or <em>åberopa</em> obstacles to <em>verkställighetshinder</em>. There may also exist a <em>verkställighetshinder</em>, as in <em>föreligga verkställighetshinder</em>.</p>
<p>The word is typically followed by the word combination <em>för utvisning</em> but it can also be <em>av skyddskäl</em>, which should also be included in a dictionary article. Consequently, the findings in Kubord 2 make both the description of the semantics of the words and its phraseological behavior more comprehensive.</p>
</section>
<section id="finding-the-data-set" class="level3">
<h3 class="anchored" data-anchor-id="finding-the-data-set">Finding the data set</h3>
<p>The data sets can be downloaded from the Språkbanken Text (<a href="https://spraakbanken.gu.se/resurser/kubord">Kubord 1</a> and <a href="https://spraakbanken.gu.se/resurser/kubord2">Kubord 2</a>) or accessed through Korp’s Kubord mode <a href="https://spraakbanken.gu.se/korp/?mode=kubord">Korp’s Kubord mode</a>. You can also conveniently search and explore the data through <a href="https://ws.spraakbanken.gu.se/docs/korp">Korp’s API</a>.</p>
<p>Kubord offers intriguing material for humanistic research, which would otherwise only be accessible to researchers within library premises due to copyright protection. While the context of sentences in which words occur is not available in these data sets, the annotated word frequencies and syntactically co-occurring words can still be useful for researchers interested in tracking and analyzing trends in new Swedish words.</p>
<p><strong>With this in mind, there is a wealth of data waiting to be explored. Feel free to dive in and explore it yourself!</strong></p>
</section>
<section id="acknowledgements" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="acknowledgements">Acknowledgements</h3>
<p>Part of this development work was carried out within <a href="https://www.humi%20nfra.se/">HUMINFRA</a> infrastructure project.</p>



<div class="no-row-height column-margin column-container"><div class="">
<div class="quarto-figure quarto-figure-left">
<figure class="figure">
<p><img src="https://kb-labb.github.io/posts/2023-08-29-kubord/huminfra.svg" class="img-fluid quarto-figure quarto-figure-left figure-img" style="width:35.0%" alt="HUMINFRA"></p>
</figure>
</div>
</div></div></section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{forsberg2023,
  author = {Forsberg, Markus and Sikora, Justyna and Sköldberg, Emma},
  title = {Words Unboxed: Discovering New Words with {Kubord}},
  date = {2023-08-29},
  url = {https://kb-labb.github.io/posts/2023-08-29-kubord/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-forsberg2023" class="csl-entry quarto-appendix-citeas">
Forsberg, Markus, Justyna Sikora, and Emma Sköldberg. 2023. <span>“Words
Unboxed: Discovering New Words with Kubord.”</span> August 29, 2023. <a href="https://kb-labb.github.io/posts/2023-08-29-kubord/">https://kb-labb.github.io/posts/2023-08-29-kubord/</a>.
</div></div></section></div> ]]></description>
  <guid>https://kb-labb.github.io/posts/2023-08-29-kubord/</guid>
  <pubDate>Mon, 28 Aug 2023 22:00:00 GMT</pubDate>
  <media:content url="https://kb-labb.github.io/posts/2023-08-29-kubord/kubord_prev.png" medium="image" type="image/png" height="88" width="144"/>
</item>
<item>
  <title>For how long is a person recognisable by their voice?</title>
  <dc:creator>Maya Nachesa</dc:creator>
  <link>https://kb-labb.github.io/posts/2023-07-04-for-how-long-is-a-person-recognisable-by-their-voice/</link>
  <description><![CDATA[ 





<p>Speaker verification is the computational task of indicating whether two audio recordings (each containing only one person’s speech) come from the same person or not. This comparison is done by converting the audio recordings to voiceprints, an abstract representation of someone’s voice. Recent approaches have used deep-learning models to create these voiceprints. TitaNet has been a particularly successful model, which was trained on the speaker identification task (who is speaking), and, once it achieved satisfactory results, used to create voiceprints.</p>

<div class="no-row-height column-margin column-container"><div class="">
<p><strong>Editor’s note:</strong> This is a blog post summarizing a master’s thesis project. A part of this project was carried out at KBLab. Maya’s full thesis can be reached at <a href="https://urn.kb.se/resolve?urn=urn:nbn:se:uu:diva-503403">the following link</a> <span class="citation" data-cites="Nachesa1762445">(Nachesa 2023)</span>.</p>
</div></div><p>Riksdagen has released a public database with all of its speeches, who was speaking when, and what they said. However, this data is not always accurate. Thus, it can be interesting to use the identity of the speakers where the data is accurate to identify them where it’s less accurate. But people are members of parliament for years. The speaker verification and identification tasks are usually performed in a smaller time-frame, and so it is unclear how the results will extend to when there are larger age-gaps between two voiceprints. Besides searching a database, this is also important when using someone’s voice to unlock, for instance, a device. It is important to know what the range is, so the user can be warned in due time when it is time to record a new voiceprint.</p>
<section id="objective" class="level2">
<h2 class="anchored" data-anchor-id="objective">Objective</h2>
<p>The main objective of this project was to investigate to what extent speakers remain recognisable by their voice as the age-gap of the speaker between the current and comparison recording increases. Further goals were to examine the effect of the age at which the comparison recording was made, the length of the recordings used for the comparisons, and the gender of the speakers.</p>
</section>
<section id="data" class="level2">
<h2 class="anchored" data-anchor-id="data">Data</h2>
<p>The data for this project consisted of Riksdagen’s parliamentary speeches. In the blog “Finding Speeches in the Riksdag’s debates” <span class="citation" data-cites="rekathati2023finding">(Rekathati 2023a)</span> you can read about how the speeches were segmented and their precise timestamps were determined, and in the blog “RixVox: A Swedish Speech Corpus with 5500 Hours of Speech from Parliamentary Debates” <span class="citation" data-cites="rekathati2023rixvox">(Rekathati 2023b)</span> you can read about the resulting dataset.</p>
<section id="extracting-speeches" class="level3">
<h3 class="anchored" data-anchor-id="extracting-speeches">Extracting speeches</h3>
<p>Sometimes speeches contained speakers other than the main one at the beginning and at the end of the audio file. Because of this, the first and last 10 seconds of each speech were excluded. Then, for each speech, the audio was extracted at 7 different lengths, namely at 1, 3, 5, 10, 30 and 60 seconds, and also at the full speech length. To ensure that it was possible to extract 60 seconds from each speech, only those speeches with a length of at least 80 seconds (to account for excluding the first and last 10 seconds) were used. Each of the segments was extracted from a random point, and only once for each speech.</p>
</section>
<section id="filtering-the-data" class="level3">
<h3 class="anchored" data-anchor-id="filtering-the-data">Filtering the data</h3>
<!-- rewrite the sentence about exclusing speeches because of speakers not mentioning themselves -->
<p>The data was also filtered for other characteristics. First, I excluded speeches containing speakers that did not have a birthyear or speaker ID associated with them. For some speeches, it seemed to be the case that a speaker mentioned themselves in a speech. However, manual inspection of the recordings revealed that the speeches had simply been assigned the wrong speaker ID, and that it was another speaker mentioning the speaker in question. These speeches were also removed Additionally, I excluded speeches that were doubled, and where the content and speech ID was swapped, but not the speaker ID. This latter case otherwise often resulted in an audio segment being attributed to the wrong person.</p>
<p>Finally, the speeches were only included if they met the following criteria: First, their <code>length_ratio</code> and <code>overlap_ratio</code> was between 0.7 and 1.3. The <code>length_ratio</code> indicates how much longer the segment predicted by diarisation was compared to its length as predicted by ASR, and the <code>overlap_ratio</code> indicates how much overlap there is in terms of time between the segment predicted by diarisation and ASR. In addition to this, a speech was only included if it was associated with one segment as per the diarisation. Finally, for each speaker, I only included them and their speeches if they had at least 3 speeches in a year (that could potentially all be from the same debate).</p>
</section>
<section id="converting-to-voiceprints" class="level3">
<h3 class="anchored" data-anchor-id="converting-to-voiceprints">Converting to voiceprints</h3>
<p>The next step was to convert the extracted audio to voiceprints. For this I used <code>TitaNet-large</code> <span class="citation" data-cites="koluguri2022titanet">(Koluguri, Park, and Ginsburg 2022)</span>. TitaNet was trained on the speaker identification task, which meant its final layer was the size of the number of speakers it was trained to recognise. The way this model was trained, meant that the representations in the previous layer maximise the cosine similarity when they belong to different speakers, and minise it when they belong to the same speaker. It is this 192-dimensional 1D vector that is extracted as the voiceprint of a speaker. Some of the speeches overloaded the GPU, so these were excluded.</p>
</section>
<section id="division-of-data" class="level3">
<h3 class="anchored" data-anchor-id="division-of-data">Division of data</h3>
<p>The data was further divided into train, dev, and test. The test data is created first. I tested how voiceprints age for a total age-gap of 9 years, and so only kept those speakers that were active in each of those years from the start of their presence in parliament. After this, I bucketed the speakers into age-ranges of 5 years. Each bucket contained a maximum of 4 speakers (balanced for gender, unless not possible). After this, I paired up the voiceprints in 4 different manners. They can be first distinguished by whether they compare the same or a different speaker, and then by whether they compare them at the same age or different age. The table below shows what the age-gaps are for each of the combinations.</p>
<table class="caption-top table">
<caption>Age-gaps for the 4 data groups</caption>
<thead>
<tr class="header">
<th>pairs \ age</th>
<th>same age</th>
<th>different age</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><strong>same speaker</strong></td>
<td>same age</td>
<td>0-9 years difference</td>
</tr>
<tr class="even">
<td><strong>different speaker</strong></td>
<td>max 5 years difference</td>
<td>random age-gap</td>
</tr>
</tbody>
</table>
<p>The train and dev data were created by first excluding the speakers already included in the test. Then, they were also bucketed in age-ranges of 5 years, but I put no requirements on for how long they had to be active in parliament. The data between train and dev had an approximate 80:20 ratio. However, where a bucket in the dev data contained fewer than 4 speakers, those speakers were instead added to the train, and the dev bucket remained empty. Then, the same 4 pairings as for the test were made for the train and dev data. See the below table for a full description of the data for train, dev, and test.</p>
<table class="caption-top table">
<caption>Data distribution and characteristics per split</caption>
<colgroup>
<col style="width: 57%">
<col style="width: 14%">
<col style="width: 14%">
<col style="width: 14%">
</colgroup>
<thead>
<tr class="header">
<th>Measure</th>
<th>Train</th>
<th>Dev</th>
<th>Test</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>First debate</td>
<td>2003-11-11</td>
<td>2004-01-23</td>
<td>2006-01-25</td>
</tr>
<tr class="even">
<td>Last debate</td>
<td>2023-02-03</td>
<td>2023-01-31</td>
<td>2021-12-08</td>
</tr>
<tr class="odd">
<td>Number of debates</td>
<td>3156</td>
<td>743</td>
<td>1191</td>
</tr>
<tr class="even">
<td>Number of speeches</td>
<td>7422</td>
<td>1363</td>
<td>2310</td>
</tr>
<tr class="odd">
<td>Number of speakers</td>
<td>177</td>
<td>36</td>
<td>20</td>
</tr>
<tr class="even">
<td>Youngest age</td>
<td>19</td>
<td>24</td>
<td>24</td>
</tr>
<tr class="odd">
<td>Oldest age</td>
<td>78</td>
<td>68</td>
<td>58</td>
</tr>
<tr class="even">
<td>Debates per year, mean (std.)</td>
<td>353 (222)</td>
<td>68 (32)</td>
<td>144 (75)</td>
</tr>
<tr class="odd">
<td>Lowest number of debates (year)</td>
<td>6 (2003)</td>
<td>7 (2023)</td>
<td>15 (2021)</td>
</tr>
<tr class="even">
<td>Highest number of debates (year)</td>
<td>694 (2016)</td>
<td>123 (2015)</td>
<td>238 (2016)</td>
</tr>
<tr class="odd">
<td>Number of speeches per speaker, mean (std.)</td>
<td>42 (44)</td>
<td>38 (32)</td>
<td>116 (36)</td>
</tr>
</tbody>
</table>
<p>Finally, I grouped the data along the length of the audio used to create the voiceprints. I created 8 groups in total. 7 of the groups paired up voiceprints extracted from pairs with the same source audio length. In other words, the group of length 1 only contained pairs where both source audios were 1 second long. The final group contained pairs comparing all source audio length combinations. This meant it compared pairs of voiceprints coming from 1 and 3 second long audio, 30 and 10, two full speeches, and so forth. The first 7 groups are referred to by their length, while this last group is referred to as “all”.</p>
</section>
</section>
<section id="method" class="level2">
<h2 class="anchored" data-anchor-id="method">Method</h2>
<section id="voiceprint-separability" class="level3">
<h3 class="anchored" data-anchor-id="voiceprint-separability">Voiceprint separability</h3>
<p>The first thing I examined, was whether the voiceprints were separable at all. That is, can they be grouped by different speakers? For this, I used T-SNE to create a graph of all the voiceprints used. If speeches cluster together by speaker, this indicates that there are likely many similarities between them. One drawback is that if some speeches end up behind another group, we will not be able to see this.</p>
</section>
<section id="setting-a-threshold" class="level3">
<h3 class="anchored" data-anchor-id="setting-a-threshold">Setting a threshold</h3>
<p>To perform speaker verification, we need some way to compare two voiceprints to each other. For voiceprints from TitaNet, this is done by computing the cosine similarity between two voiceprints. A score closer to 1 indicates that the voiceprints are very similar, and likely belong to the same person, while a score closer to -1 (or 0 depending on how it’s calculated) indicates they are very dissimilar, and likely belong to different people. However, only having this score is not enough for us to know whether two voiceprints belong to the same speaker or not. To this end, we can set a threshold: For every score above or equal to this threshold we say that the two voiceprints come from the same speaker, and otherwise they come from two different speakers. This threshold however, needs to be determined. This is where the training and dev data comes in. If we set the threshold on the same data we are testing on, we cannot be sure that the results we get are due to this threshold and the resulting aging of the voiceprints is generalisable. Because of this, I set the threshold on the training data. I use the dev data to verify that, no matter where I set the threshold, the accuracy scores between the training and dev scores are going to be similar.</p>
<p>To set a threshold, we need to know what the scores for the same speaker group and different speaker group look like, to be able to distinguish them. For the same-speaker group, I used the same-age division of data. This is for two reasons. First: in real-world applications, speaker recordings are likely to be made in a short period of time, thereby not varying greatly in age. Second, I am testing what the effect is of testing voiceprints against different ages, meaning we cannot already include the effect of the aging voice when setting the threshold. For the different-speaker group, I used the different-age division, as we do not care to distinguish between different speakers of the same age, but about being able to tell the difference between different speakers at all. The threshold is set using <code>sklearn</code>’s <code>roc_curve</code>, and <code>scipy</code>’s <code>interp1d</code> and <code>brentq</code>.</p>
</section>
<section id="investigating-the-effects-of-age-segment-length-and-gender" class="level3">
<h3 class="anchored" data-anchor-id="investigating-the-effects-of-age-segment-length-and-gender">Investigating the effects of age, segment length, and gender</h3>
<p>Thresholds are set and tested using the “all” speech lengths group, unless stated otherwise.</p>
<section id="age" class="level4">
<h4 class="anchored" data-anchor-id="age">Age</h4>
<p>I investigated the effects of age in two ways. First, I tested the effect of an increase in the age-gap on the accuracy, False Positive Rate (FPR), and False Negative Rate (FNR). In addition to this, I also investigated the effect of when the first voiceprint was recorded. In other words, if a voiceprint is recorded for someone at 29 years of age and then used for the next 9 years, how does this compare to when it is, for instance, recorded at 39 years of age and used for the next 9? Do we have a more stable voice in certain spans of life than others?</p>
</section>
<section id="segment-length" class="level4">
<h4 class="anchored" data-anchor-id="segment-length">Segment length</h4>
<p>To test the effect of segment length, I split the “all” speech lengths group among the comparisons, to investigate the effect of each individual speech length on the accuracy, FPR, and FNR. In addition to this, I also set the threshold for each of the 8 speech length groups, to investigate how this affects the height of the threshold set, and in turn how that affects the accuracy.</p>
</section>
<section id="gender" class="level4">
<h4 class="anchored" data-anchor-id="gender">Gender</h4>
<p>I investigated the effect of gender on how the voiceprint ages. In other words, do voices age differently for men and women?</p>
</section>
</section>
</section>
<section id="results-and-discussion" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="results-and-discussion">Results and discussion</h2>
<div id="fig-tsne" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-tsne-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="images/T-SNE_for_all_speeches_at_all_speech_lengths.png" class="lightbox" data-gallery="quarto-lightbox-gallery-1" title="Figure&nbsp;1: T-SNE plot of test speakers at all lengths"><img src="https://kb-labb.github.io/posts/2023-07-04-for-how-long-is-a-person-recognisable-by-their-voice/images/T-SNE_for_all_speeches_at_all_speech_lengths.png" class="img-fluid figure-img"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-tsne-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;1: T-SNE plot of test speakers at all lengths
</figcaption>
</figure>
</div>
<p>Figure&nbsp;1 shows the T-SNE graph for all the test speakers. As we can see, the speakers generally group together among themselves, indicating that, despite age differences and varying lengths of the speech data used to create the voiceprints, they are still globally recognisable as belonging to the same person. Nevertheless, we see that there are also a few speeches that end up in vastly different places, suggesting that their representation was not as robust as the rest of the group.</p>
<div class="page-columns page-full">
<div id="fig-trainacc" class="quarto-layout-panel page-columns page-full">
<figure class="quarto-float quarto-float-fig figure page-columns page-full">
<div aria-describedby="fig-trainacc-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca" class="page-columns page-full">
<div class="quarto-layout-row column-body-outset">
<div class="quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="fig-trainacc" style="flex-basis: 50.0%;justify-content: flex-start;">
<div id="fig-trainaccleft" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-fig figure">
<div aria-describedby="fig-trainaccleft-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="images/train_acc_vs_fnr_n_fpr_within_age_within_age_VS_across_speaker_all.svg" class="lightbox" data-gallery="fig-trainacc" title="Figure&nbsp;2&nbsp;(a): Accuracy, FPR, and FNR at different thresholds"><img src="https://kb-labb.github.io/posts/2023-07-04-for-how-long-is-a-person-recognisable-by-their-voice/images/train_acc_vs_fnr_n_fpr_within_age_within_age_VS_across_speaker_all.svg" class="img-fluid figure-img" data-ref-parent="fig-trainacc"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-subfloat-caption quarto-subfloat-fig" id="fig-trainaccleft-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(a) Accuracy, FPR, and FNR at different thresholds
</figcaption>
</figure>
</div>
</div>
<div class="quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="fig-trainacc" style="flex-basis: 50.0%;justify-content: flex-start;">
<div id="fig-trainaccright" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-fig figure">
<div aria-describedby="fig-trainaccright-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="images/train_VS_dev_acc_within_age_VS_across_speaker_all_speech_lengths.svg" class="lightbox" data-gallery="fig-trainacc" title="Figure&nbsp;2&nbsp;(b): Comparison of train and dev accuracy"><img src="https://kb-labb.github.io/posts/2023-07-04-for-how-long-is-a-person-recognisable-by-their-voice/images/train_VS_dev_acc_within_age_VS_across_speaker_all_speech_lengths.svg" class="img-fluid figure-img" style="width:91.0%" data-ref-parent="fig-trainacc"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-subfloat-caption quarto-subfloat-fig" id="fig-trainaccright-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(b) Comparison of train and dev accuracy
</figcaption>
</figure>
</div>
</div>
</div>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-trainacc-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;2: Accuracy, FPR, and FNR at different thresholds.
</figcaption>
</figure>
</div>
</div>
<p>Figure&nbsp;2 shows the training accuracy, FNR, and FPR (<strong>left plot</strong>), and compares the training and dev accuracy (<strong>right plot</strong>). The vertical dashed line indicates the threshold. The threshold is set at 0.463, and we can see that there is no large difference in accuracy between the train and dev. This gives us confidence that the threshold we set is generalisable.</p>
<div id="fig-cossims-all" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-cossims-all-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="images/train_all_cossim_score.svg" class="lightbox" data-gallery="quarto-lightbox-gallery-4" title="Figure&nbsp;3: Cosine similarity scores for 3 groups: same-speaker same-age, same-speaker different-age, and different-speaker different-age."><img src="https://kb-labb.github.io/posts/2023-07-04-for-how-long-is-a-person-recognisable-by-their-voice/images/train_all_cossim_score.svg" class="img-fluid figure-img"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-cossims-all-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;3: Cosine similarity scores for 3 groups: same-speaker same-age, same-speaker different-age, and different-speaker different-age.
</figcaption>
</figure>
</div>
<p>Figure&nbsp;3 shows the cosine similarity scores for the two same-speaker groups and the different-speaker different-age group. As we can see, the cosine similarity scores are generally higher when comparing the voiceprints of one speaker against each other. What we also see, however, is that when we introduce an age-gap between the voiceprints, that the cosine similarity scores drop slightly. This suggests that the voice aging does become slightly more dissimilar compared to the first recording.</p>
<div class="page-columns page-full">
<div id="fig-testacc" class="quarto-layout-panel page-columns page-full">
<figure class="quarto-float quarto-float-fig figure page-columns page-full">
<div aria-describedby="fig-testacc-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca" class="page-columns page-full">
<div class="quarto-layout-row column-body-outset">
<div class="quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="fig-testacc" style="flex-basis: 50.0%;justify-content: flex-start;">
<div id="fig-testaccleft" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-fig figure">
<div aria-describedby="fig-testaccleft-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="images/test_within_speaker_across_age_VS_across_speaker_all_speech_lengths_cossim_score.svg" class="lightbox" data-gallery="fig-testacc" title="Figure&nbsp;4&nbsp;(a): Percentiles (at intervals of 10%) of cosine similarities for each age-gap, and comparing two different speakers."><img src="https://kb-labb.github.io/posts/2023-07-04-for-how-long-is-a-person-recognisable-by-their-voice/images/test_within_speaker_across_age_VS_across_speaker_all_speech_lengths_cossim_score.svg" class="img-fluid figure-img" data-ref-parent="fig-testacc"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-subfloat-caption quarto-subfloat-fig" id="fig-testaccleft-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(a) Percentiles (at intervals of 10%) of cosine similarities for each age-gap, and comparing two different speakers.
</figcaption>
</figure>
</div>
</div>
<div class="quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="fig-testacc" style="flex-basis: 50.0%;justify-content: flex-start;">
<div id="fig-testaccright" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-fig figure">
<div aria-describedby="fig-testaccright-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="images/test_acc_vs_fnr_n_fpr_across_age_VS_across_speaker_all_speech_lengths.svg" class="lightbox" data-gallery="fig-testacc" title="Figure&nbsp;4&nbsp;(b): Test accuracy, FNR, and FPR for each age-gap."><img src="https://kb-labb.github.io/posts/2023-07-04-for-how-long-is-a-person-recognisable-by-their-voice/images/test_acc_vs_fnr_n_fpr_across_age_VS_across_speaker_all_speech_lengths.svg" class="img-fluid figure-img" data-ref-parent="fig-testacc"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-subfloat-caption quarto-subfloat-fig" id="fig-testaccright-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(b) Test accuracy, FNR, and FPR for each age-gap.
</figcaption>
</figure>
</div>
</div>
</div>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-testacc-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;4: Effect of age-gap on cosine similarity, accuracy, FNR, and FPR scores.
</figcaption>
</figure>
</div>
</div>
<p>Figure&nbsp;4 shows how the cosine similarity scores develop as the age-gap between two voiceprints increases (Figure&nbsp;4 (a)), and also when comparing the voiceprints of two different speakers. In general, the cosine similarity seems to drop as the age-gap increases, but the cosine similarity is much lower when comparing two different speakers. The Figure&nbsp;4 (b) in the same figure shows how, as the age-gap between two voiceprints increases, the accuracy drops slightly, and experiences an even sharper drop around the 5-year age-gap. This is characterised by an increase in FNR as the age-gap increases.</p>
<p>In general, it seems to be the case that increasing the age-gap between two voiceprints does affect our ability to recognise that they come from the same speaker, and so some caution needs to be exercised when using someone’s voiceprint to recognise them at a very different point in time.</p>
<div class="page-columns page-full">
<div id="fig-startageacc" class="quarto-layout-panel page-columns page-full">
<figure class="quarto-float quarto-float-fig figure page-columns page-full">
<div aria-describedby="fig-startageacc-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca" class="page-columns page-full">
<div class="quarto-layout-row column-body-outset">
<div class="quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="fig-startageacc" style="flex-basis: 50.0%;justify-content: flex-start;">
<div id="fig-startageaccleft" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-fig figure">
<div aria-describedby="fig-startageaccleft-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="images/test_bucket_acc_vs_fnr_n_fpr_within_within_age_VS_across_speaker_all_speech_lengths.svg" class="lightbox" data-gallery="fig-startageacc" title="Figure&nbsp;5&nbsp;(a): Effect of starting age on same-age accuracy."><img src="https://kb-labb.github.io/posts/2023-07-04-for-how-long-is-a-person-recognisable-by-their-voice/images/test_bucket_acc_vs_fnr_n_fpr_within_within_age_VS_across_speaker_all_speech_lengths.svg" class="img-fluid figure-img" data-ref-parent="fig-startageacc"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-subfloat-caption quarto-subfloat-fig" id="fig-startageaccleft-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(a) Effect of starting age on same-age accuracy.
</figcaption>
</figure>
</div>
</div>
<div class="quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="fig-startageacc" style="flex-basis: 50.0%;justify-content: flex-start;">
<div id="fig-startageaccright" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-fig figure">
<div aria-describedby="fig-startageaccright-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="images/test_bucket_acc_vs_fnr_n_fpr_across_across_age_VS_across_speaker_all_speech_lengths.svg" class="lightbox" data-gallery="fig-startageacc" title="Figure&nbsp;5&nbsp;(b): Effect of starting age on accuracy for a 9 year age-span."><img src="https://kb-labb.github.io/posts/2023-07-04-for-how-long-is-a-person-recognisable-by-their-voice/images/test_bucket_acc_vs_fnr_n_fpr_across_across_age_VS_across_speaker_all_speech_lengths.svg" class="img-fluid figure-img" data-ref-parent="fig-startageacc"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-subfloat-caption quarto-subfloat-fig" id="fig-startageaccright-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(b) Effect of starting age on accuracy for a 9 year age-span.
</figcaption>
</figure>
</div>
</div>
</div>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-startageacc-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;5: Effect of starting age on same-age and different-age scores.
</figcaption>
</figure>
</div>
</div>
<p>Figure&nbsp;5 shows that speakers in the 29-33 age-range are the easiest to recognise, and older and younger speakers are harder to recognise (Figure&nbsp;5 (a)). However, the difference is not large. The Figure&nbsp;5 (b) uses different-age group for the same-speaker group. That is, the speaker verification is tested for this entire 9-year age-range for each speaker. We see that accuracy is highest when someone’s voiceprint is recorded in the 29-33 age-range and used for the next 9 years, and that it is lower for other age groups. Presumably this could be the case due to younger speakers’ voices still changing too much, and older speakers’ having begun to change again.</p>
<div id="fig-heatmap-acc" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-heatmap-acc-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="images/test_accuracy_heatmap.svg" class="lightbox" data-gallery="quarto-lightbox-gallery-9" title="Figure&nbsp;6: Heatmap of FNR and FPR scores for all speech length comparisons. Left plot: Heatmap of FNR for all speech length comparisons. Right plot: Heatmap of FPR for all speech length comparisons."><img src="https://kb-labb.github.io/posts/2023-07-04-for-how-long-is-a-person-recognisable-by-their-voice/images/test_accuracy_heatmap.svg" class="img-fluid figure-img"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-heatmap-acc-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;6: Heatmap of FNR and FPR scores for all speech length comparisons. <strong>Left plot</strong>: Heatmap of FNR for all speech length comparisons. <strong>Right plot</strong>: Heatmap of FPR for all speech length comparisons.
</figcaption>
</figure>
</div>
<p>Figure&nbsp;6 shows that accuracy is very low when at least one of the two voiceprints in a pair comes from a 1-second long audio. It also shows that accuracy is slightly diminished when the pairs use longer audio.</p>
<div class="page-columns page-full">
<div id="fig-heatmap" class="quarto-layout-panel page-columns page-full">
<figure class="quarto-float quarto-float-fig figure page-columns page-full">
<div aria-describedby="fig-heatmap-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca" class="page-columns page-full">
<div class="quarto-layout-row column-body-outset">
<div class="quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="fig-heatmap" style="flex-basis: 50.0%;justify-content: flex-start;">
<div id="fig-fnr-heatmap" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-fig figure">
<div aria-describedby="fig-fnr-heatmap-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="images/test_fnr_heatmap.svg" class="lightbox" data-gallery="fig-heatmap" title="Figure&nbsp;7&nbsp;(a): Heatmap of FNR for all speech length comparisons."><img src="https://kb-labb.github.io/posts/2023-07-04-for-how-long-is-a-person-recognisable-by-their-voice/images/test_fnr_heatmap.svg" class="img-fluid figure-img" data-ref-parent="fig-heatmap"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-subfloat-caption quarto-subfloat-fig" id="fig-fnr-heatmap-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(a) Heatmap of FNR for all speech length comparisons.
</figcaption>
</figure>
</div>
</div>
<div class="quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="fig-heatmap" style="flex-basis: 50.0%;justify-content: flex-start;">
<div id="fig-fpr-heatmap" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-fig figure">
<div aria-describedby="fig-fpr-heatmap-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="images/test_fpr_heatmap.svg" class="lightbox" data-gallery="fig-heatmap" title="Figure&nbsp;7&nbsp;(b): Heatmap of FPR for all speech length comparisons."><img src="https://kb-labb.github.io/posts/2023-07-04-for-how-long-is-a-person-recognisable-by-their-voice/images/test_fpr_heatmap.svg" class="img-fluid figure-img" data-ref-parent="fig-heatmap"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-subfloat-caption quarto-subfloat-fig" id="fig-fpr-heatmap-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(b) Heatmap of FPR for all speech length comparisons.
</figcaption>
</figure>
</div>
</div>
</div>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-heatmap-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;7: Heatmap of FNR and FPR scores for all speech length comparisons.
</figcaption>
</figure>
</div>
</div>
<p>Figure&nbsp;7 sheds some light on the accuracy scores. The very low performance of the short speeches seems to be attributable to a high FNR (Figure&nbsp;7 (a)), meaning that we were much more likely to miss when two voiceprints belonged to the same person. The lower performance for longer audio lengths is attributable to a reduction in FPR (Figure&nbsp;7 (b)): we were more likely to accidentally mark two unrelated speakers as being the same person.</p>
<p>Given all this, it might seem best to use speeches at around 3 seconds long: after all, these gave the best results, right? But that does not paint the whole story.</p>
<div class="page-columns page-full">
<div id="fig-threshold" class="quarto-layout-panel page-columns page-full">
<figure class="quarto-float quarto-float-fig figure page-columns page-full">
<div aria-describedby="fig-threshold-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca" class="page-columns page-full">
<div class="quarto-layout-row column-page-inset">
<div class="quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="fig-threshold" style="flex-basis: 50.0%;justify-content: flex-start;">
<div id="fig-thresholdleft" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-fig figure">
<div aria-describedby="fig-thresholdleft-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="images/test_acc_n_thresholds_per_speech_length.svg" class="lightbox" data-gallery="fig-threshold" title="Figure&nbsp;8&nbsp;(a): The effect of speech length on threshold and the subsequent accuracy."><img src="https://kb-labb.github.io/posts/2023-07-04-for-how-long-is-a-person-recognisable-by-their-voice/images/test_acc_n_thresholds_per_speech_length.svg" class="img-fluid figure-img" style="width:85.0%" data-ref-parent="fig-threshold"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-subfloat-caption quarto-subfloat-fig" id="fig-thresholdleft-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(a) The effect of speech length on threshold and the subsequent accuracy.
</figcaption>
</figure>
</div>
</div>
<div class="quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="fig-threshold" style="flex-basis: 50.0%;justify-content: flex-start;">
<div id="fig-thresholdright" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-fig figure">
<div aria-describedby="fig-thresholdright-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="images/test_within_speaker_across_age_all_speech_lengths_means_cossim_score.svg" class="lightbox" data-gallery="fig-threshold" title="Figure&nbsp;8&nbsp;(b): The effect of speech length and the age-gap on cosine similarity. Ranges represent 95% confidence intervals."><img src="https://kb-labb.github.io/posts/2023-07-04-for-how-long-is-a-person-recognisable-by-their-voice/images/test_within_speaker_across_age_all_speech_lengths_means_cossim_score.svg" class="img-fluid figure-img" style="width:100.0%" data-ref-parent="fig-threshold"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-subfloat-caption quarto-subfloat-fig" id="fig-thresholdright-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(b) The effect of speech length and the age-gap on cosine similarity. Ranges represent 95% confidence intervals.
</figcaption>
</figure>
</div>
</div>
</div>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-threshold-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;8: The effect of speech length on the threshold and cosine similarity.
</figcaption>
</figure>
</div>
</div>
<p>Figure&nbsp;8 shows how, as we increase the audio length from which the voiceprints were extracted, the threshold is set higher as well (Figure&nbsp;8 (a)). However, the corresponding accuracy is quite high, especially for voiceprint pairs extracted from speeches ranging from 5 seconds long to full-length speeches. The biggest dip in performance can be seen for voiceprint pairs extracted from 1-second long audio, and for “all” speech-length comparisons. When looking at Figure&nbsp;8 (b), we see that the cosine similarity increases for both the same-speaker and different-speaker voiceprint pairs, although moreso for the former group. It seems that voiceprint quality increases as the audio length from which they were extracted increases, resulting in the need to set a higher threshold to distinguish between the two groups. This also explains the reduced performance for the shortest audio lengths and for the mixed audio-length comparisons. For the short audio, the cosine similarities between the same- and different-speaker groups are considerably closer to each other, resulting in a larger overlap, and thereby worse performance. For the “all” group, the low threshold is still too high to correctly classify the voiceprints coming from 1-second long audio, but too low to correctly distinguish voiceprint pairs coming from longer audio. The overall impression these two graphs seem to give is that it is not necessarily that one needs longer audio for more accurate speaker verification, but rather that the threshold should be set and tested on voiceprints coming from audio of the same length.</p>
<div id="fig-gender" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-gender-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="images/test_gender_acc_across_across_age_VS_across_speaker_all_speech_lengths.svg" class="lightbox" data-gallery="quarto-lightbox-gallery-14" title="Figure&nbsp;9: The effect of gender on speaker verification."><img src="https://kb-labb.github.io/posts/2023-07-04-for-how-long-is-a-person-recognisable-by-their-voice/images/test_gender_acc_across_across_age_VS_across_speaker_all_speech_lengths.svg" class="img-fluid figure-img"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-gender-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;9: The effect of gender on speaker verification.
</figcaption>
</figure>
</div>
<p>Figure&nbsp;9 shows that as the age-gap between two voiceprints for the same speaker increases, the speaker verification drops at different rates for men and women after roughly 5 years, suggesting that it decreases more strongly for men.</p>
</section>
<section id="conclusion-and-future-research" class="level2">
<h2 class="anchored" data-anchor-id="conclusion-and-future-research">Conclusion and future research</h2>
<p>All in all, the results from this research suggests that voiceprints do age, and experience a sharper drop after the age-gap between two voiceprints reaches about 5 years. If a voiceprint is recorded between 29-33 years of age and used for the next 9, speaker verification retains a higher accuracy than if it is recorded at a different age. Using longer audio results in higher quality voiceprints, but it is also important to simply use and set thresholds on audio coming from the same length. However, mixing audio-length combinations still yields strong accuracy scores. Using very short audio yielded poor results, possibly due to the audio being extracted at random points from the speeches. Finally, male voiceprints might age faster than female voiceprints.</p>
<p>One thing that should be highlighted in this investigation, is that each age group was very small. Future research should endeavour to use larger groups to solidify the results obtained. Additionally, future research could increase the age-gap investigated, to see whether the general trend continues beyond this range. It would also be good to investigate both younger and older speakers. Additionally, we saw that voiceprints remain stable at different rates when recorded at different starting ages. It would be interesting to investigate what this range looks like for these age groups. Finally, given that speaker verification is quite accurate for shorter age-ranges, it would be interesting to combine the voiceprint from multiple ages to see whether that extends the number of years for which speaker verification remains accurate.</p>
</section>


<div id="quarto-appendix" class="default"><section id="code" class="level2 appendix"><h2 class="anchored quarto-appendix-heading">Code</h2><div class="quarto-appendix-contents">

<p>The code for this project can be found at <a href="https://github.com/MKNachesa/masters_thesis">https://github.com/MKNachesa/masters_thesis</a>.</p>



</div></section><section class="quarto-appendix-contents" id="quarto-bibliography"><h2 class="anchored quarto-appendix-heading">References</h2><div id="refs" class="references csl-bib-body hanging-indent" data-entry-spacing="0">
<div id="ref-koluguri2022titanet" class="csl-entry">
Koluguri, Nithin Rao, Taejin Park, and Boris Ginsburg. 2022. <span>“TitaNet: Neural Model for Speaker Representation with 1D Depth-Wise Separable Convolutions and Global Context.”</span> In <em>ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)</em>, 8102–6. IEEE.
</div>
<div id="ref-Nachesa1762445" class="csl-entry">
Nachesa, Maya Konstantinovna. 2023. <span>“How Do Voiceprints Age?”</span> Master’s thesis, Uppsala University, Department of Linguistics; Philology; Uppsala University, Department of Linguistics; Philology. <a href="https://www.diva-portal.org/smash/record.jsf?pid=diva2:1762445">https://www.diva-portal.org/smash/record.jsf?pid=diva2:1762445</a>.
</div>
<div id="ref-rekathati2023finding" class="csl-entry">
Rekathati, Faton. 2023a. <span>“The KBLab Blog: Finding Speeches in the Riksdag’s Debates.”</span> <a href="https://kb-labb.github.io/posts/2023-02-15-finding-speeches-in-the-riksdags-debates/">https://kb-labb.github.io/posts/2023-02-15-finding-speeches-in-the-riksdags-debates/</a>.
</div>
<div id="ref-rekathati2023rixvox" class="csl-entry">
———. 2023b. <span>“The KBLab Blog: RixVox: A Swedish Speech Corpus with 5500 Hours of Speech from Parliamentary Debates.”</span> <a href="https://kb-labb.github.io/posts/2023-03-09-rixvox-a-swedish-speech-corpus/">https://kb-labb.github.io/posts/2023-03-09-rixvox-a-swedish-speech-corpus/</a>.
</div>
</div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{nachesa2023,
  author = {Nachesa, Maya},
  title = {For How Long Is a Person Recognisable by Their Voice?},
  date = {2023-07-04},
  url = {https://kb-labb.github.io/posts/2023-07-04-for-how-long-is-a-person-recognisable-by-their-voice/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-nachesa2023" class="csl-entry quarto-appendix-citeas">
Nachesa, Maya. 2023. <span>“For How Long Is a Person Recognisable by
Their Voice?”</span> July 4, 2023. <a href="https://kb-labb.github.io/posts/2023-07-04-for-how-long-is-a-person-recognisable-by-their-voice/">https://kb-labb.github.io/posts/2023-07-04-for-how-long-is-a-person-recognisable-by-their-voice/</a>.
</div></div></section></div> ]]></description>
  <guid>https://kb-labb.github.io/posts/2023-07-04-for-how-long-is-a-person-recognisable-by-their-voice/</guid>
  <pubDate>Mon, 03 Jul 2023 22:00:00 GMT</pubDate>
  <media:content url="https://kb-labb.github.io/posts/2023-07-04-for-how-long-is-a-person-recognisable-by-their-voice/images/tsne2.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>A robust, multi-label sentiment classifier for Swedish</title>
  <dc:creator>Hillevi Hägglöf</dc:creator>
  <link>https://kb-labb.github.io/posts/2023-06-16-a-robust-multi-label-sentiment-classifier-for-swedish/</link>
  <description><![CDATA[ 





<p>Many researchers in the humanities and adjacent fields are interested in the tonality of texts, for which sentiment analysis is an excellent tool. KBLab presents a robust, transformer based sentiment classifier in Swedish. The model is available as a multi-class model (negative/neutral/positive). It is publicly available via <a href="https://huggingface.co/KBLab/robust-swedish-sentiment-multiclass">the Hugging Face Hub</a>, published under the Apache 2.0 license.</p>
<p>The model was developed in collaboration with KBLab researcher Nora Hansson Bittár, who is a PhD student at the Stockholm School of Economics. She is currently studying the development of sentiments and emotional load in the Swedish media landscape over time, inspired by Rozado et al (2022)<sup>1</sup>. Nora’s project will be further documented in the blog.</p>
<p>A particular requirement when studying tonality in news media, is the need of a category representing a neutral tone, as many news articles are neither inherently positive or negative. This is often overlooked in sentiment modeling, as it adds complexity to the task, which in turn affects model performance.</p>
<p>Another aspect of other available sentiment models available in the Swedish language that makes them difficult to use in a project like Nora’s is their poor generalization capabilities. Most, if not all, previously published sentiment models in Swedish are trained on one type of text exclusively (reviews), which consequently leads to poor performance in other linguistic domains. We have trained our models on multiple datasets of various types and sizes.</p>
<p>Robustness, in this case, refers to a language model’s generalization capabilities. Since most, if not all, previously published sentiment models in Swedish are trained on only one type of text (reviews), the performance in other linguistic domains suffer. We have trained our models on five different datasets from different sources, of various sizes and quality. Note that these datasets do not have a consistent, underlying annotation schema. This is compensated by the relatively large size of the corpus.</p>
<ul>
<li>13K Trustpilot reviews. This dataset is based on <a href="https://github.com/timpal0l/ScandiSent">ScandiSent</a> and includes reviews of products and services, along with a star rating of 1-5. The original dataset, however, excluded reviews with a star rating of 3, which are the best neutral proxies. Because of this, we have added to the corpus by scraping neutral reviews. The resulting <a href="https://github.com/gilleti/trustpilot-scraper">Trustpilot scraper</a> is freely available.</li>
<li>12K Twitter posts, manually annotated by Niklas Palm (included with Niklas’ permission). The dataset is available on <a href="https://github.com/niklas-palm/sentiment-classification/">Github</a>.</li>
<li>4K news headlines, manually annotated at KBLab. This dataset cannot be shared due to copyright restrictions of the source material.</li>
<li>5K texts from <a href="https://spraakbanken.gu.se/resurser/absabank-imm">Språkbankens ABSAImm corpus</a> on the topic of immigration. Data is collected from news and social media.</li>
<li>40K of machine translated reviews from <a href="https://github.com/ltgoslo/norec">the Norwegian Review Corpus (NoReC)</a>. Texts were translated using CTranslate2 with a <a href="https://github.com/Helsinki-NLP/Tatoeba-Challenge/tree/master/models/nor-swe">translation model from Helsinki NLP</a>. The translations have not been formally evaluated but experiments show that they do in fact contribute positively to the sentiment classification.</li>
</ul>
<p>The accuracy of the model is evaluated on a balanced test set and is measured at 0.80 for the multiclass version and 0.88 for the binary version. More extensive evaluation will be conducted at a later stage and included in Nora’s report (albeit not on a balanced test set, but in the news media domain specifically). Both models are finetuned on the <a href="https://huggingface.co/KBLab/megatron-bert-large-swedish-cased-165k">Swedish BERT-large model with 340M parameters</a>, developed here at KBLab.</p>
<section id="inference" class="level1 page-columns page-full">
<h1>Inference</h1>
<p>Below is a minimal example showcasing how to perform predictions with the model using the transformers pipeline.</p>
<div id="22d16982" class="cell" data-execution_count="1">
<div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb1-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> transformers <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> AutoTokenizer, AutoModelForSequenceClassific</span>
<span id="cb1-2">ation, pipeline</span>
<span id="cb1-3"></span>
<span id="cb1-4">tokenizer <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> AutoTokenizer.from_pretrained(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"KBLab/megatron-bert-large-swedish-cased-165k"</span>)</span>
<span id="cb1-5">model <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> AutoModelForSequenceClassification.from_pretrained(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"KBLab/robust-swedish-sentiment-multiclass"</span>)</span>
<span id="cb1-6"></span>
<span id="cb1-7">text <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Rihanna uppges gravid"</span></span>
<span id="cb1-8">classifier <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> pipeline(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sentiment-analysis"</span>, model<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>model, tokenizer<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>tokenizer)</span>
<span id="cb1-9">result <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> classifier(text)</span></code></pre></div>
</div>
<p>In this case, the classifier outputs the label NEUTRAL with a score of 0.89. Good luck and happy inference!</p>
<section id="acknowledgements" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="acknowledgements">Acknowledgements</h2>
<p>Part of this development work was carried out within <a href="https://www.humi%20nfra.se/">HUMINFRA</a> infrastructure project.</p>

<div class="no-row-height column-margin column-container"><div class="">
<div class="quarto-figure quarto-figure-left">
<figure class="figure">
<p><img src="https://kb-labb.github.io/posts/2023-06-16-a-robust-multi-label-sentiment-classifier-for-swedish/huminfra.svg" class="img-fluid figure-img" style="width:35.0%"></p>
<figcaption>HUMINFRA</figcaption>
</figure>
</div>
</div></div><p>Preview photo by <a href="http://libris.kb.se/bib/22447938">Bo Wingård (1967)</a>.</p>


</section>
</section>


<div id="quarto-appendix" class="default"><section id="footnotes" class="footnotes footnotes-end-of-document"><h2 class="anchored quarto-appendix-heading">Footnotes</h2>

<ol>
<li id="fn1"><p>Rozado D, Hughes R, Halberstadt J (2022) Longitudinal analysis of sentiment and emotion in news media headlines using automated labeling with Transformer language models. PLoS ONE 17(10): e0276367. <a href="https://doi.org/10.1371/journal.pone.0276367">https://doi.org/10.1371/journal.pone.0276367</a>↩︎</p></li>
</ol>
</section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{hägglöf2023,
  author = {Hägglöf, Hillevi},
  title = {A Robust, Multi-Label Sentiment Classifier for {Swedish}},
  date = {2023-06-16},
  url = {https://kb-labb.github.io/posts/2023-06-16-a-robust-multi-label-sentiment-classifier-for-swedish/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-hägglöf2023" class="csl-entry quarto-appendix-citeas">
Hägglöf, Hillevi. 2023. <span>“A Robust, Multi-Label Sentiment
Classifier for Swedish.”</span> June 16, 2023. <a href="https://kb-labb.github.io/posts/2023-06-16-a-robust-multi-label-sentiment-classifier-for-swedish/">https://kb-labb.github.io/posts/2023-06-16-a-robust-multi-label-sentiment-classifier-for-swedish/</a>.
</div></div></section></div> ]]></description>
  <guid>https://kb-labb.github.io/posts/2023-06-16-a-robust-multi-label-sentiment-classifier-for-swedish/</guid>
  <pubDate>Thu, 15 Jun 2023 22:00:00 GMT</pubDate>
  <media:content url="https://kb-labb.github.io/posts/2023-06-16-a-robust-multi-label-sentiment-classifier-for-swedish/Wingard_Bo_Dagen_H_3_KoB_F_Saml_36.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Swedish speech synthesis</title>
  <dc:creator>Faton Rekathati</dc:creator>
  <link>https://kb-labb.github.io/posts/2023-05-24-swedish-text-to-speech/</link>
  <description><![CDATA[ 





<p>Creating realistic sounding speech with proper intonation, pitch and tone from text has long been a goal of speech synthesis systems. These systems have a wide range of applications, among which a few are:</p>
<ol type="1">
<li>Accessibility tooling for the visually impaired via screen readers.</li>
<li>Helping people with speech impairments communicate.</li>
<li>Allowing for interaction with computers, digital assistants, and robots through audio as opposed to text.</li>
</ol>
<p>Recent developments in neural speech synthesis have allowed for the synthetization of voices with increasing natural fidelity. However, many of these high quality systems with support for smaller languages remain proprietary.</p>
<p>At KBLab we recently discovered a relatively user friendly option to train a neural speech synthesis model through the Piper library<sup>1</sup>. Luckily, the Norwegian Language Bank (Språkbanken) maintains several speech datasets originally produced by the company NST (Nordisk Språkteknologi). One of those datasets consists of 5300 recordings of a single Swedish speaker <sup>2</sup>, recorded for the purposes of training speech synthesis systems.</p>
<section id="have-a-listen" class="level2">
<h2 class="anchored" data-anchor-id="have-a-listen">Have a listen</h2>
<p>Have a listen to the output of KBLab’s model below. We use text from the wikipedia articles on <a href="https://sv.wikipedia.org/wiki/Regnb%C3%A5ge">Regnbåge</a> and <a href="https://sv.wikipedia.org/wiki/Europaparlamentet">Europaparlementet</a> as the source text. For each sample, we also compare our model to a recently released open source text-to-speech model from Meta AI. See the Massively Multilingual Speech project (MMS) <sup>3</sup> for further details about Meta’s model.</p>
<div class="fullbox">
  <div class="audiobox">
    <div class="metabox">
      <h3 class="anchored">KBLab TTS (Piper)</h3>
      <audio controls="" style="width: 290px;">
         <source src="audio/kblab_regnbage.wav" type="audio/wav">
      </audio>
    </div>
  </div>
  
  <div class="audiobox">
    <div class="metabox2">
      <h3 class="anchored">Facebook/Meta TTS</h3>
      <audio controls="" style="width: 290px;">
        <source src="audio/meta_mms_regnbage.wav" type="audio/wav">
      </audio>
    </div>
  </div>
</div>
<p>&nbsp;</p>
<blockquote class="blockquote">
<p>En regnbåge är ett optiskt, meteorologiskt fenomen som uppträder som ett (nästintill) fullständigt ljusspektrum i form av en båge på himlen då solen lyser på nedfallande regn. Regnbågen består färgmässigt av en kontinuerlig övergång från rött (ytterst) via gula, gröna och blå nyanser till violett innerst; ofta definieras antalet färger som sju, inklusive orange och indigo.</p>
</blockquote>
<div class="fullbox2">
  <div class="audiobox">
    <div class="metabox">
      <h3 class="anchored">KBLab TTS (Piper)</h3>
    <audio controls="" style="width: 290px;">
      <source src="audio/kblab_eu.wav" type="audio/wav">
    </audio>
    </div>
  </div>
  
  <div class="audiobox">
    <div class="metabox2">
      <h3 class="anchored">Facebook/Meta TTS</h3>
      <audio controls="" style="width: 290px;">
        <source src="audio/meta_mms_eu.wav" type="audio/wav">
      </audio>
    </div>
  </div>
</div>
<p>&nbsp;</p>
<blockquote class="blockquote">
<p>Europaparlamentet (EP), även känt som EU-parlamentet, är den ena lagstiftande institutionen inom Europeiska unionen; den andra är Europeiska unionens råd. Parlamentet, som består av 705 ledamöter, väljs genom allmänna och direkta val vart femte år, och företräder unionsmedborgarna direkt på unionsnivå. Parlamentet kan förenklat liknas vid ett underhus i ett tvåkammarsystem.</p>
</blockquote>
<section id="odd-pronounciations" class="level3">
<h3 class="anchored" data-anchor-id="odd-pronounciations">Odd pronounciations</h3>
<p>Piper and the MMS model from Meta both use <a href="https://github.com/jaywalnut310/vits">VITS</a> to train the text-to-speech model. This model relies on <a href="https://en.wikipedia.org/wiki/ESpeak">espeak-ng</a> to translate text to phonemes. The extent and comprehensiveness of <code>espeak-ng</code>’s pronounciation and prosody rules vary from language to language, as the software is largely reliant on volunteer contributions. The reason the words <em>meteorologiskt</em>, <em>fenomen</em> and <em>kontinuerlig</em> have such a strange pronounciations is because <code>espeak-ng</code> generates an incorrect text to phoneme conversion <sup>4</sup>.</p>
</section>
</section>
<section id="how-do-i-use-the-model" class="level2">
<h2 class="anchored" data-anchor-id="how-do-i-use-the-model">How do I use the model?</h2>
<p>To try out KBLab’s TTS model yourself using Piper:</p>
<ol type="1">
<li>Download the Piper binary <a href="https://github.com/rhasspy/piper">from Github</a> (executable file that allows you to run the model in your terminal) <sup>5</sup>. For Linux:</li>
</ol>
<div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">wget</span> https://github.com/rhasspy/piper/releases/download/v0.0.2/piper_amd64.tar.gz</span></code></pre></div>
<ol start="2" type="1">
<li>Unzip/Untar the downloaded archive in a directory of your choice.</li>
</ol>
<div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb2-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># The contents will be untarred to directory named piper/</span></span>
<span id="cb2-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tar</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-xvf</span> piper_amd64.tar.gz</span></code></pre></div>
<ol start="3" type="1">
<li>Download the Svenska (Swedish) model weights from <a href="https://rhasspy.github.io/piper-samples/">Piper’s voice samples</a>.</li>
</ol>
<div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb3-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">wget</span> https://github.com/rhasspy/piper/releases/download/v0.0.2/voice-sv-se-nst-medium.tar.gz</span></code></pre></div>
<ol start="4" type="1">
<li>Unzip/Untar the downloaded archive file in a directory of your choice. We suggest doing it in the same directory where you untarred the piper binary: <code>piper/</code>.</li>
</ol>
<div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb4-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tar</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-xvf</span> voice-sv-se-nst-medium.tar.gz <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--directory</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"piper"</span></span></code></pre></div>
<ol start="5" type="1">
<li>Generate speech via the terminal.</li>
</ol>
<div class="sourceCode" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb5-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Let's first move in to the piper directory</span></span>
<span id="cb5-2"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">cd</span> piper</span>
<span id="cb5-3"></span>
<span id="cb5-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Generate speech to the audio file min_talsyntes.wav</span></span>
<span id="cb5-5"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">echo</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'Jag genererar tal med hjälp av talsyntes.'</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">./piper</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb5-6">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--model</span> sv-se-nst-medium.onnx <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb5-7">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--output_file</span> min_talsyntes.wav</span></code></pre></div>
</section>
<section id="pretrained-model-checkpoints" class="level2">
<h2 class="anchored" data-anchor-id="pretrained-model-checkpoints">Pretrained model checkpoints</h2>
<p>For anyone interested in using this model to finetune other voices, we have uploaded the <a href="https://huggingface.co/KBLab/piper-tts-nst-swedish">pretrained checkpoint weights to Huggingface</a>.</p>
<div class="sourceCode" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb6-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">git</span> clone https://huggingface.co/KBLab/piper-tts-nst-swedish</span></code></pre></div>


</section>


<div id="quarto-appendix" class="default"><section id="footnotes" class="footnotes footnotes-end-of-document"><h2 class="anchored quarto-appendix-heading">Footnotes</h2>

<ol>
<li id="fn1"><p>https://github.com/rhasspy/piper↩︎</p></li>
<li id="fn2"><p>https://www.nb.no/sprakbanken/en/resource-catalogue/oai-nb-no-sbr-18/↩︎</p></li>
<li id="fn3"><p>https://ai.facebook.com/blog/multilingual-model-speech-recognition/↩︎</p></li>
<li id="fn4"><p>https://github.com/rhasspy/piper/issues/72#issuecomment-1550170779↩︎</p></li>
<li id="fn5"><p>https://github.com/rhasspy/piper↩︎</p></li>
</ol>
</section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{rekathati2023,
  author = {Rekathati, Faton},
  title = {Swedish Speech Synthesis},
  date = {2023-05-24},
  url = {https://kb-labb.github.io/posts/2023-05-24-swedish-text-to-speech/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-rekathati2023" class="csl-entry quarto-appendix-citeas">
Rekathati, Faton. 2023. <span>“Swedish Speech Synthesis.”</span> May 24,
2023. <a href="https://kb-labb.github.io/posts/2023-05-24-swedish-text-to-speech/">https://kb-labb.github.io/posts/2023-05-24-swedish-text-to-speech/</a>.
</div></div></section></div> ]]></description>
  <guid>https://kb-labb.github.io/posts/2023-05-24-swedish-text-to-speech/</guid>
  <pubDate>Tue, 23 May 2023 22:00:00 GMT</pubDate>
  <media:content url="https://kb-labb.github.io/posts/2023-05-24-swedish-text-to-speech/tts_banner.png" medium="image" type="image/png" height="135" width="144"/>
</item>
<item>
  <title>Scientific discourse with BERTopic</title>
  <dc:creator>Hillevi Hägglöf</dc:creator>
  <dc:creator>Justyna Sikora</dc:creator>
  <link>https://kb-labb.github.io/posts/2023-03-17-scientific-discourse-with-bertopic/</link>
  <description><![CDATA[ 





<style>
body {text-align: justify}
</style>
<p>Topic modeling is a great way to discover and characterize the contents of a large text collection. It can help us understand what themes are present in a corpus. They can uncover trends and provide insights where a human, even an expert, would struggle. There are, however, limitations to topic modeling. The main one lies in the interpretation. The results are not always easily understood. Some topics may lack quality. There might be uncategorized documents. Other times the results do not align with a researcher’s expectation or intuition. The most important thing to understand is that topic models alone do not provide all the answers to the contents of a corpus. It is simply a tool to navigate large amounts of texts, and should be used as such.</p>
<p>In this post, we will describe a typical topic modeling use case using the transformer-based BERTopic on scientific abstracts. We will do this with the specific aim of highlighting the problems that might arise during the interpretation of the results. For further interested, an introduction to BERTopic and usage with KBLab’s BERT-models can be found <a href="https://kb-labb.github.io/posts/2022-06-14-bertopic/">here</a>.</p>
<section id="the-bertopic-pipeline" class="level2">
<h2 class="anchored" data-anchor-id="the-bertopic-pipeline">The BERTopic pipeline</h2>
<p>The process of topic modeling with BERTopic is roughly as follows: collect the data → transform the data into numerical representations → reduce the dimensionality of these representations → group data points into clusters → describe the content of the clusters.</p>
<p>Here, we’re working with a collection of scientific abstracts from the research field of education. We have downloaded abstracts alongside some metadata (for example: author names, publishing years and author affiliation). The abstracts were scraped from the Science Citation Index Expanded, which is a citation index of over 9000 scientific journals included in the Web of Science (WoS). 144 567 abstracts were collected between the years 1990-2022. Since scientific abstracts are short and distinct (this specific dataset has an average word count of 200), they are an excellent candidate for topic modeling.</p>
<p>The next step of the process is to convert the abstracts into a numerical representation. There are many ways of doing this but the preferred method is by using a pre-trained BERT model to generate embeddings. These are vector representations of the texts that capture the context of words and phrases. Since the data collection was limited to abstracts from papers written in English, we are able to use one of the many pre-trained English language models. In this case, we used DistilBERT, which is a small and fast Transformer model trained by distilling the bloated BERT base. DistilBERT provides a good tradeoff between performance and speed and is a good option, especially if you want to run the model on a laptop. The resulting embeddings are designed to capture complex relationships, making them high in dimension and thus difficult to work with. In order to perform the actual topic modeling, we need to reduce the dimensionality. This is done by using UMAP, which creates a low-dimensional representation of the abstract embeddings.</p>
<p>The goal of topic modeling, simply put, is to group documents into meaningful categories that – hopefully – represent some underlying topics. BERTopic does this by applying a density-based clustering algorithm called HDBSCAN. Very briefly, HDBSCAN groups together dense regions of data points in some feature space using a minimum-spanning tree (a type of graph) in order to connect all vertices of the tree with the minimum total weight.</p>
<p>At this point in the process, we have a set of clusters – groups of documents that are similar in some way. But how do we understand those clusters? What do they actually have in common? The BERTopic pipeline solves this by applying a class-based Term Frequency-Inverse Document Frequency (TF-IDF) matrix. TF-IDF in its original form basically estimates the importance of a word in a document by comparing the frequency of a word in that document to the frequency of the same word in all other documents. The class-based TF-IDF does this on a cluster level by treating all documents in a cluster as one document. This results in a list of the top n most important words in a cluster. We let that list represent a topic.</p>
<p>Et voilà, our topic model is done. Let’s have a look at the results.</p>
</section>
<section id="the-scientific-education-discourse" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="the-scientific-education-discourse">The scientific education discourse</h2>
<p>The intertopic distance map is an interactive way of exploring the topics. It is quite intuitive – the closer the topics are on the map, the more similar they are in terms of their content. The scientific educational discourse is here represented by 107 topics (as determined automatically by BERTopic as the optimal number of topics), organized into 20 larger clusters. In general, the bigger the cluster (and the topic), the more heterogeneous that cluster or topic will be. Take for example the large cluster in the top-left quadrant of the map. This cluster holds a dozen or so subtopics where some are obviously related: topic 86 and 91, for example, are clearly covid-19 topics. These are closely related to the online learning topic 25 and 38. This is no accident. The online learning topics, however, are at an even greater proximity to topic 0, which is the largest topic in the model. This topic is represented by the words “internal”, “cultural”, “international students”, “higher education” and “expatriates”. The relationship between these topics are less clear, which emphasizes the need of thorough examination of the quality of the topics at hand.</p>
<iframe src="intertopic.html" width="100%" height="670px" frameborder="0"></iframe>
<p>Some patterns in the data are obvious, even to non-experts such as ourselves. As expected, there are several clusters of topics that represent teaching subjects: biology, medicine, accounting, physical education, programming, robotics and many more. There’s also several clusters of topics concerning various facets of being a teacher with topics characterized by words and phrases such as black teachers, teacher identity, mentors, coaching and so on. We can also identify different levels of the educational system, from schools for small children to grad school. There’s several socio-economical clusters that discuss, for example, class, intergenerational mobility, financial aid, segregation and charter schools. These are just some examples but they help give an indication that the topic model is, in fact, relatively sound. There is almost certainly more to this map that an expert could shed some light on. Interested parties are welcome to contribute.</p>
<p>Another way to get an overview of the topics and their internal relationship is by visualizing the potential hierarchical structure of the topic model. The number of topics can be overwhelming, especially since many topics are overlapping. By creating a hierarchical structure, we can easily trace the relationships between topics. As we can see in the graph, some of them are very similar, as demonstrated by for example topics 86 and 61, both concerned with covid. The two topics below, 33 and 17, are connected to medicine and nursing, which is a good fit with the articles about the pandemic. In the same branch we can also identify topics related to online learning and online discussions. Note the close proximity to the covid topics. Looking at the hierarchy can be especially useful if we would like to reduce the number of topics. The hierarchical clustering graph gives us an overview of how similar topics are and therefore can be a good indicator of how many topics could be potentially merged.</p>
<div class="page-columns page-full">
<div id="fig-hierarchy" class="quarto-float quarto-figure quarto-figure-center anchored page-columns page-full">
<figure class="quarto-float quarto-float-fig figure page-columns page-full">
<div aria-describedby="fig-hierarchy-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca" class="page-columns page-full">
<a href="hierarchy.png" class="lightbox page-columns page-full" data-gallery="quarto-lightbox-gallery-1" title="Figure&nbsp;1: Hierarchical topic structure of scientific abstracts in the field of education 1990-2022"><img src="https://kb-labb.github.io/posts/2023-03-17-scientific-discourse-with-bertopic/hierarchy.png" class="img-fluid figure-img column-body-outset"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-hierarchy-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;1: Hierarchical topic structure of scientific abstracts in the field of education 1990-2022
</figcaption>
</figure>
</div>
</div>
<p>We can also explore the topic evolution by providing timestamps for all documents present in the model. In this case, years. The plot represents the frequencies of documents within a topic for a specific year. We can also limit the number of topics we are interested in. Once again, following the steps of the covid topics, we can clearly see the emergence of massive open online courses around 2008. Due to the pandemic, the topic frequency spikes again.</p>
<div class="page-columns page-full">
<div id="fig-topics-over-time" class="quarto-float quarto-figure quarto-figure-center anchored page-columns page-full">
<figure class="quarto-float quarto-float-fig figure page-columns page-full">
<div aria-describedby="fig-topics-over-time-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca" class="page-columns page-full">
<a href="topics_over_time.png" class="lightbox page-columns page-full" data-gallery="quarto-lightbox-gallery-2" title="Figure&nbsp;2: Online learning topics over time."><img src="https://kb-labb.github.io/posts/2023-03-17-scientific-discourse-with-bertopic/topics_over_time.png" class="img-fluid figure-img column-body-outset"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-topics-over-time-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;2: Online learning topics over time.
</figcaption>
</figure>
</div>
</div>
</section>
<section id="taking-bertopic-further" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="taking-bertopic-further">Taking BERTopic further</h2>
<p>Another aspect of topic models to keep in mind as a researcher is that a topic model alone does not constitute research. The topic model provides overview and helps us navigate large quantities of text but it is only of many tools in the quest for new knowledge. It might shine a new light on a dataset but in order to actually produce new knowledge, further analysis is needed. We provide an example of this below by correlating the topics with metadata found in the dataset, namely the nationality of authors. (This variable is not easily available in the dataset but can be extracted through some text wrangling methods that we will not elaborate further upon here.) With information not only about topics but also author affiliation at hand, we can examine what type of research emerges from specific cultural contexts. We do this by performing a correspondence analysis (CA) on our enriched topic model. Correspondence analysis (CA) is a statistical method for analyzing categorical data, conceptually similar to the more well-known principal components analysis (PCA). Using CA, we can visualize associations between topics and affiliations. The analysis is performed using <a href="https://github.com/MaxHalford/prince">Prince</a>, a Python library for multivariate exploratory data analysis in Python. To reduce the amount of data points, only the 50 most well-represented countries in the dataset are included in the graph. The graph is still very busy, indicating that further reduction of data might be necessary.</p>
<div class="page-columns page-full">
<div id="fig-ca" class="quarto-float quarto-figure quarto-figure-center anchored page-columns page-full">
<figure class="quarto-float quarto-float-fig figure page-columns page-full">
<div aria-describedby="fig-ca-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca" class="page-columns page-full">
<a href="ca_education_50_countries.png" class="lightbox page-columns page-full" data-gallery="quarto-lightbox-gallery-3" title="Figure&nbsp;3: Correspondence analysis visualization over topics and author affiliation."><img src="https://kb-labb.github.io/posts/2023-03-17-scientific-discourse-with-bertopic/ca_education_50_countries.png" class="img-fluid figure-img column-screen-inset"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-ca-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;3: Correspondence analysis visualization over topics and author affiliation.
</figcaption>
</figure>
</div>
</div>
<p>Before looking at the plot, there are some things to take into consideration. Interpreting the results of a CA is not exactly straightforward and a more in-depth guide can be found <a href="https://www.displayr.com/interpret-correspondence-analysis-plots-probably-isnt-way-think/">here</a>, but the most important things to know is that (1) datapoints close to the origin are less distinct than data points further away and (2) a small angle between a topic and affiliation, indicates an association. The interpretation of the actual plotted data should ideally be done by a domain expert to be meaningful. We’ll give it a go nonetheless.</p>
<p>In the left-lower quadrant, we can see that China is closely related to the covid-19 topics, which, to our knowledge, aligns well with the general Chinese discourse. China also appears to have a negative association to the USA, we can be found in the right-lower quadrant. USA stands out as a distinct scientific discourse in the right-lower quadrant, with very subject focused topics such as STEM, psychology and engineering. Other finds, that at least partially, validates our analysis is a cluster of nordic countries in the upper-right quadrant. This indicates a shared scientific educational discourse. Some topics that are distinct to the nordic countries are PISA, reflective practices and ethical values.</p>
</section>
<section id="conclusions" class="level2">
<h2 class="anchored" data-anchor-id="conclusions">Conclusions</h2>
<p>In this post, we have revisited BERTopic and provided some useful tools to mitigate the limitations of the method. Demonstrated here is a non-exhaustive list of ways to validate your topic model, as well as examples on how to perform further explorative data analysis using BERTopic as a base.</p>
</section>
<section id="acknowledgements" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="acknowledgements">Acknowledgements</h2>
<p>Part of this development work was carried out within <a href="https://www.huminfra.se/">HUMINFRA</a> infrastructure project.</p>



<div class="no-row-height column-margin column-container"><div class="">
<div class="quarto-figure quarto-figure-left">
<figure class="figure">
<div class="quarto-figure quarto-figure-left">
<figure class="figure">
<p><a href="huminfra.svg" class="lightbox" data-gallery="quarto-lightbox-gallery-4" title="HUMINFRA"><img src="https://kb-labb.github.io/posts/2023-03-17-scientific-discourse-with-bertopic/huminfra.svg" class="img-fluid quarto-figure quarto-figure-left figure-img" style="width:35.0%" alt="HUMINFRA"></a></p>
</figure>
</div>
<figcaption>HUMINFRA</figcaption>
</figure>
</div>
</div></div></section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{hägglöf2023,
  author = {Hägglöf, Hillevi and Sikora, Justyna},
  title = {Scientific Discourse with {BERTopic}},
  date = {2023-03-30},
  url = {https://kb-labb.github.io/posts/2023-03-17-scientific-discourse-with-bertopic/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-hägglöf2023" class="csl-entry quarto-appendix-citeas">
Hägglöf, Hillevi, and Justyna Sikora. 2023. <span>“Scientific Discourse
with BERTopic.”</span> March 30, 2023. <a href="https://kb-labb.github.io/posts/2023-03-17-scientific-discourse-with-bertopic/">https://kb-labb.github.io/posts/2023-03-17-scientific-discourse-with-bertopic/</a>.
</div></div></section></div> ]]></description>
  <guid>https://kb-labb.github.io/posts/2023-03-17-scientific-discourse-with-bertopic/</guid>
  <pubDate>Wed, 29 Mar 2023 22:00:00 GMT</pubDate>
  <media:content url="https://kb-labb.github.io/posts/2023-03-17-scientific-discourse-with-bertopic/Forskarsalen_1920-talet.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>RixVox: A Swedish Speech Corpus with 5500 Hours of Speech from Parliamentary Debates</title>
  <dc:creator>Faton Rekathati</dc:creator>
  <link>https://kb-labb.github.io/posts/2023-03-09-rixvox-a-swedish-speech-corpus/</link>
  <description><![CDATA[ 





<p>Automatic Speech Recognition (ASR) systems that convert spoken language to text rely heavily on annotated data to produce the best possible results. Such datasets are unfortunately not widely available for Swedish. The combined total of currently available audio datasets with annotated transcripts for the Swedish language number somewhere in the hundreds of hours.</p>
<p>To this end, KBLab releases Rixvox, a new Swedish ASR dataset consisting of <img src="https://latex.codecogs.com/png.latex?5500"> hours of speech. The data originates from parliamentary debates between the years of <img src="https://latex.codecogs.com/png.latex?2003"> to <img src="https://latex.codecogs.com/png.latex?2023">, which were made available via the Swedish Parliament’s open data initiative. KBLab used written protocols to segment speeches from the debates, and to subsequently force align the the written transcripts with audio from the speeches. In addition to audio and transcripts, metadata such as the name, gender, birth year, political party, and electoral district of speakers is also available.</p>
<p>RixVox is free and open for anyone to download and use. The dataset can be reached on the following link: https://huggingface.co/datasets/KBLab/rixvox .</p>
<section id="rixvox-dataset-statistics" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="rixvox-dataset-statistics">RixVox dataset statistics</h2>
<p>The RixVox dataset was constructed from parliamentary debates. You can read more about how we segmented speeches from debates and determined their precise start and end location within a debate in a previous article on this blog: “Finding Speeches in the Riksdag’s debates” <span class="citation" data-cites="rekathati2023finding">(Rekathati 2023)</span>.</p>
<p>The dataset has chunked the audio from speeches in to smaller snippets suitable for training ASR models. Each observation is <em>up to</em> 30 seconds in length, and consists of either a single or several sentences from the written transcript of a speech, along with the corresponding audio. The dataset consists of a total of <img src="https://latex.codecogs.com/png.latex?5493.6"> hours of speech. There are <img src="https://latex.codecogs.com/png.latex?1194"> different speakers represented in the data. The average duration of an observation is <img src="https://latex.codecogs.com/png.latex?23.68"> seconds. In the table below, we present the distribution of observations of the different train, validation and test split of RixVox, along with some summary statistics for each split.</p>
<div class="column-body">
<table class="caption-top table">
<colgroup>
<col style="width: 11%">
<col style="width: 15%">
<col style="width: 28%">
<col style="width: 27%">
<col style="width: 16%">
</colgroup>
<thead>
<tr class="header">
<th>Dataset Split</th>
<th style="text-align: right;">Observations</th>
<th style="text-align: right;">Total duration of speech (hours)</th>
<th style="text-align: right;">Average duration obs. (seconds)</th>
<th style="text-align: right;">Number of speakers</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Train</td>
<td style="text-align: right;">818227</td>
<td style="text-align: right;">5383</td>
<td style="text-align: right;">23.69</td>
<td style="text-align: right;">1165</td>
</tr>
<tr class="even">
<td>Validation</td>
<td style="text-align: right;">7933</td>
<td style="text-align: right;">52</td>
<td style="text-align: right;">23.50</td>
<td style="text-align: right;">18</td>
</tr>
<tr class="odd">
<td>Test</td>
<td style="text-align: right;">8884</td>
<td style="text-align: right;">59</td>
<td style="text-align: right;">23.74</td>
<td style="text-align: right;">11</td>
</tr>
</tbody>
</table>
</div>
<p>The dataset splits were created by sampling speakers until a threshold was reached in terms of total duration of speech. For the training set, we randomly sampled speakers until <img src="https://latex.codecogs.com/png.latex?98%5C%25"> of the total duration of the RixVox dataset was reached (<img src="https://latex.codecogs.com/png.latex?5384"> hours). For the test and validation set, we randomly sampled speakers until each filled up to a bucket of <img src="https://latex.codecogs.com/png.latex?1%5C%25"> of the total duration of the entire dataset.</p>
<p>Let’s also take a look at the gender distribution of speakers. We have <img src="https://latex.codecogs.com/png.latex?602"> men, <img src="https://latex.codecogs.com/png.latex?519"> women, and <img src="https://latex.codecogs.com/png.latex?73"> speakers for whom this metadata is missing.</p>
<div data-pagedtable="false">
<script data-pagedtable-source="" type="application/json">
{"columns":[{"label":["Gender"],"name":[1],"type":["chr"],"align":["left"]},{"label":["Number"],"name":[2],"type":["int"],"align":["right"]}],"data":[{"1":"female","2":"519"},{"1":"male","2":"602"},{"1":"NA","2":"73"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}}
  </script>
</div>
<p>Looking at the total duration of speech for each gender, we have a similar distribution to above. <img src="https://latex.codecogs.com/png.latex?46.3%5C%25"> of the individual speakers were women, and <img src="https://latex.codecogs.com/png.latex?44.3%5C%25"> of the total duration of speeches in RixVox is made up of women speaking.</p>
<div class="column-body">
<div data-pagedtable="false">
<script data-pagedtable-source="" type="application/json">
{"columns":[{"label":["Gender"],"name":[1],"type":["chr"],"align":["left"]},{"label":["Total duration (hours)"],"name":[2],"type":["dbl"],"align":["right"]}],"data":[{"1":"female","2":"2361.15"},{"1":"male","2":"2967.57"},{"1":"NA","2":"164.92"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}}
  </script>
</div>
</div>
<section id="most-and-least-intelligible-electoral-districts" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="most-and-least-intelligible-electoral-districts">Most and least intelligible electoral districts</h3>
<p>Each observation in our dataset belongs to a speech in a debate. After segmenting the speeches from debate audio files, we machine transcribed every speech using KBLab’s <a href="https://huggingface.co/KBLab/wav2vec2-large-voxrex-swedish">wav2vec2-large-voxrex-swedish</a> model <span class="citation" data-cites="wav2vec2">(Malmsten, Haffenden, and Börjeson 2022)</span>. We then calculated the <a href="https://en.wikipedia.org/wiki/BLEU">BLEU score</a> to measure the correspondence between the machine generated transcription and the official written transcript. A high BLEU score indicates there’s a higher correspondence, or overlap, between the machine generated transcript and the official transcript. This may indicate that ASR systems find certain regions easier to transcribe, or may alternatively indicate that the people who transcribe the speeches tend to rephrase or reword written transcripts of speeches from these districts.</p>
<div class="column-body-outset">
<div data-pagedtable="false">
<script data-pagedtable-source="" type="application/json">
{"columns":[{"label":["Electoral district"],"name":[1],"type":["chr"],"align":["left"]},{"label":["Mean BLEU score"],"name":[2],"type":["dbl"],"align":["right"]},{"label":["Std. dev. BLEU"],"name":[3],"type":["dbl"],"align":["right"]},{"label":["Number of speakers"],"name":[4],"type":["int"],"align":["right"]}],"data":[{"1":"NA","2":"0.6000797","3":"0.1252977","4":"NA"},{"1":"Jönköpings län","2":"0.5940009","3":"0.1277287","4":"37"},{"1":"Södermanlands län","2":"0.5935326","3":"0.1206110","4":"38"},{"1":"Uppsala län","2":"0.5929714","3":"0.1270916","4":"43"},{"1":"Västra Götalands läns södra","2":"0.5898193","3":"0.1230608","4":"20"},{"1":"Stockholms län","2":"0.5854009","3":"0.1268066","4":"137"},{"1":"Skåne läns norra och östra","2":"0.5853696","3":"0.1305330","4":"32"},{"1":"Jämtlands län","2":"0.5826683","3":"0.1252455","4":"17"},{"1":"Hallands län","2":"0.5820507","3":"0.1225573","4":"38"},{"1":"Kronobergs län","2":"0.5816702","3":"0.1067194","4":"20"},{"1":"Östergötlands län","2":"0.5810539","3":"0.1335815","4":"45"},{"1":"Göteborgs kommun","2":"0.5796593","3":"0.1256837","4":"60"},{"1":"Västra Götalands läns norra","2":"0.5787399","3":"0.1258094","4":"29"},{"1":"Stockholms kommun","2":"0.5767339","3":"0.1243611","4":"126"},{"1":"Västerbottens län","2":"0.5724071","3":"0.1149559","4":"36"},{"1":"Blekinge län","2":"0.5721980","3":"0.1351392","4":"15"},{"1":"Örebro län","2":"0.5699883","3":"0.1232872","4":"33"},{"1":"Kalmar län","2":"0.5664890","3":"0.1302711","4":"28"},{"1":"Skåne läns västra","2":"0.5648055","3":"0.1237373","4":"30"},{"1":"Västmanlands län","2":"0.5628068","3":"0.1415991","4":"30"},{"1":"Norrbottens län","2":"0.5624232","3":"0.1463259","4":"26"},{"1":"Dalarnas län","2":"0.5617315","3":"0.1218160","4":"41"},{"1":"Västra Götalands läns östra","2":"0.5575997","3":"0.1214828","4":"25"},{"1":"Värmlands län","2":"0.5571123","3":"0.1232013","4":"35"},{"1":"Västra Götalands läns västra","2":"0.5566830","3":"0.1549344","4":"38"},{"1":"Västernorrlands län","2":"0.5565349","3":"0.1585068","4":"27"},{"1":"Gävleborgs län","2":"0.5548522","3":"0.1156745","4":"30"},{"1":"Skåne läns södra","2":"0.5393078","3":"0.1289290","4":"49"},{"1":"Gotlands län","2":"0.5383960","3":"0.1534656","4":"7"},{"1":"Malmö kommun","2":"0.5338347","3":"0.1100739","4":"30"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}}
  </script>
</div>
</div>
<p>The speakers with the highest score are for those whom thedistrict is missing (<code>NA</code>). These are mostly government ministers who have never been members of parliament. The least intelligible electoral districts are southern Skåne, Gotland, and Malmö municipality (also southern Skåne).</p>
</section>
<section id="longest-total-duration-speaker" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="longest-total-duration-speaker">Longest total duration speaker</h3>
<p>Which speakers have spent the most time on the Riksdag Chamber’s podium? The table below shows that Morgan Johansson is the undisputed <img src="https://latex.codecogs.com/png.latex?%5C#1"> debater in terms of total duration of speech.</p>
<div class="column-body-outset">
<div data-pagedtable="false">
<script data-pagedtable-source="" type="application/json">
{"columns":[{"label":["Speaker"],"name":[1],"type":["chr"],"align":["left"]},{"label":["Party"],"name":[2],"type":["chr"],"align":["left"]},{"label":["Total duration (hours)"],"name":[3],"type":["dbl"],"align":["right"]}],"data":[{"1":"Morgan Johansson","2":"S","3":"72.72"},{"1":"Jens Holm","2":"V","3":"61.79"},{"1":"Beatrice Ask","2":"M","3":"48.50"},{"1":"Anders Borg","2":"M","3":"47.91"},{"1":"Mikael Damberg","2":"S","3":"46.83"},{"1":"Tomas Eneroth","2":"S","3":"44.82"},{"1":"Magdalena Andersson","2":"S","3":"39.60"},{"1":"Peter Hultqvist","2":"S","3":"38.23"},{"1":"Per Bolund","2":"MP","3":"33.94"},{"1":"Monica Green","2":"S","3":"32.60"},{"1":"Anders Ygeman","2":"S","3":"31.61"},{"1":"Hillevi Larsson","2":"S","3":"31.57"},{"1":"Lena Hallengren","2":"S","3":"30.92"},{"1":"Göran Hägglund","2":"KD","3":"30.62"},{"1":"Ylva Johansson","2":"S","3":"30.36"},{"1":"Annika Strandhäll","2":"S","3":"29.15"},{"1":"Maud Olofsson","2":"C","3":"27.86"},{"1":"Ibrahim Baylan","2":"S","3":"27.51"},{"1":"Jan Björklund","2":"L","3":"27.14"},{"1":"Eva-Lena Jansson","2":"S","3":"26.82"},{"1":"Lars Tysklind","2":"L","3":"26.79"},{"1":"Sven Otto Littorin","2":"M","3":"26.61"},{"1":"Catharina Elmsäter-Svärd","2":"M","3":"26.59"},{"1":"Maria Larsson","2":"KD","3":"26.59"},{"1":"Roger Haddad","2":"L","3":"25.15"},{"1":"Börje Vestlund","2":"S","3":"24.95"},{"1":"Håkan Svenneling","2":"V","3":"24.10"},{"1":"Ulla Andersson","2":"V","3":"23.87"},{"1":"Jan Lindholm","2":"MP","3":"23.77"},{"1":"Ann Linde","2":"S","3":"23.46"},{"1":"Ola Johansson","2":"C","3":"23.44"},{"1":"Kerstin Lundgren","2":"C","3":"22.89"},{"1":"Gunnar Andrén","2":"L","3":"22.60"},{"1":"Margot Wallström","2":"S","3":"22.55"},{"1":"Patrik Björck","2":"S","3":"22.51"},{"1":"Sven-Erik Bucht","2":"S","3":"22.24"},{"1":"Eskil Erlandsson","2":"C","3":"22.04"},{"1":"Johan Hultberg","2":"M","3":"21.73"},{"1":"Isak From","2":"S","3":"21.68"},{"1":"Karin Rågsjö","2":"V","3":"21.42"},{"1":"Fredrik Malm","2":"L","3":"21.30"},{"1":"Thomas Morell","2":"SD","3":"20.81"},{"1":"Lars Beckman","2":"M","3":"20.57"},{"1":"Anna Johansson","2":"S","3":"20.34"},{"1":"Mats Odell","2":"KD","3":"20.21"},{"1":"Hillevi Engström","2":"M","3":"20.18"},{"1":"Finn Bengtsson","2":"M","3":"19.87"},{"1":"Isabella Lövin","2":"MP","3":"19.44"},{"1":"Nina Lundström","2":"L","3":"19.43"},{"1":"Ali Esbati","2":"V","3":"19.22"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}}
  </script>
</div>
</div>
</section>
</section>
<section id="method-of-creation" class="level2">
<h2 class="anchored" data-anchor-id="method-of-creation">Method of creation</h2>
<p>Before RixVox could be created, we needed to accurately segment speeches from debates. In other words: locate where the speech started and ended within the debate audio file. The most cumbersome parts of the preliminary work undertaken to segment speeches from debates is described in our previous article “Finding Speeches in the Riksdag’s Debates” <span class="citation" data-cites="rekathati2023finding">(Rekathati 2023)</span>. We recommend reading this article for background on the speech segmentation.</p>
<section id="quality-filtering" class="level3">
<h3 class="anchored" data-anchor-id="quality-filtering">Quality filtering</h3>
<p>Once the speeches were segmented, the remaining work consisted of performing some quality filtering based on simple heuristics, aligning the written transcripts with the audio on a sentence level, adding metadata about the speakers, and finally converting the alignments to short snippets up to 30 seconds in length (a suitable format for training ASR models).</p>
<p>The first round of quality filters applied on speeches can be found in the <a href="https://github.com/kb-labb/riksdagen_anforanden/blob/d0f066ed23b66eba55cc01c0b7cb5393db94e914/scripts/heuristic_filter.py#L21-L45">following lines of code</a>. These include:</p>
<ul>
<li>We keep only speeches <img src="https://latex.codecogs.com/png.latex?%3E%2025"> seconds in duration as predicted by speaker diarization (see the linked article for context). The reliability of our speech segmentation method improves with speech length.</li>
<li>We keep only speeches <img src="https://latex.codecogs.com/png.latex?%3E%2015"> seconds in duration as predicted by fuzzy string matching between machine transcription and official transcripts.</li>
<li>We calculate a “length ratio”, which is the predicted duration of the speech by speaker diarization, divided by the predicted duration of the speech by fuzzy string matching. We only keep the speech if this length ratio is between <img src="https://latex.codecogs.com/png.latex?0.8"> and <img src="https://latex.codecogs.com/png.latex?1.5">. Otherwise, we deem our two methods to be in too much of a disagreement.</li>
<li>We calculate an “overlap ratio”, which is the “duration where speaker diarization and fuzzy string matching predictions overlap” divided by the total predicted duration of the fuzzy string matching method. If this ratio is <img src="https://latex.codecogs.com/png.latex?%3E0.8"> we keep the speech.</li>
<li>We only keep speeches where <img src="https://latex.codecogs.com/png.latex?1"> single speaker was identified as speaking within the predicted regions.</li>
<li>We only keep speeches where the difference in predicted start time between a future and previous speech is <img src="https://latex.codecogs.com/png.latex?%3E5"> seconds.</li>
</ul>
<p>The second round of quality filters were applied after another fuzzy string matching sanity check was performed. This time, instead of fuzzy string matching the text of a written transcript against the machine transcription of an entire debate, we fuzzy string match the text of the written transcript against the machine transcription of the segmented speech, as predicted by our speaker diarization. A short summary of the second round of quality filters follows:</p>
<ul>
<li>Removing speeches where the official written protocol starts matching the machine transcription only after a threshold of <img src="https://latex.codecogs.com/png.latex?X"> words in the machine transcribed version. This was a possible indication that the speech segmentation had predicted a <strong>too early</strong> start location for the speech, erroneously including parts of other speakers.<br>
</li>
<li>Removing speeches where the official written protocol stops matching hte machine transcription too early.</li>
<li>We adjust the <img src="https://latex.codecogs.com/png.latex?X"> threshold based on different dates the debates were held on, and whether the debate was the first and/or the last speech of the debate (the first and last speeches of debates were more likely to be cut off in the middle of the speech before the year <img src="https://latex.codecogs.com/png.latex?2012">).</li>
</ul>
<p>See the <a href="https://github.com/kb-labb/riksdagen_anforanden/blob/d0f066ed23b66eba55cc01c0b7cb5393db94e914/scripts/rixvox_filter.py#L28-L51">following lines of code</a> for a full list of the filtering conditions.</p>
<p>The above filtering procedures reduced the number of speeches to be included in RixVox from about 122k speeches to 115k speeches.</p>
</section>
<section id="forced-alignment" class="level3">
<h3 class="anchored" data-anchor-id="forced-alignment">Forced alignment</h3>
<p>Once we had high confidence in the remaining set of predictions, we proceeded to align the written protocols with the audio. This was done by:</p>
<ol type="1">
<li>Sentence tokenizing the written transcripts.</li>
<li>Using the <a href="https://www.readbeyond.it/aeneas/"><code>aeneas</code> library</a> to force align the audio with the text on the sentence level.</li>
</ol>
<p>The <code>aeneas</code> library gives an output in the form of predicing the <code>start</code> and <code>end</code> location of the sentence within the speech.</p>
<p>We can recommend reading the masters thesis <strong>“Automatic Annotation of Speech: Exploring Boundaries within Forced Alignment for Swedish and Norwegian”</strong> <span class="citation" data-cites="Biczysko1674281">(Biczysko 2022)</span> for an excellent review of available forced alignment tools for Swedish and Norwegian.</p>
</section>
<section id="creating-30s-observation-snippets." class="level3">
<h3 class="anchored" data-anchor-id="creating-30s-observation-snippets.">Creating 30s observation snippets.</h3>
<p>In the final step, we concatenate sentences from the same speech that follow one another up to a maximum length of <img src="https://latex.codecogs.com/png.latex?30"> seconds per observation. The observations in RixVox are thus composed of either a single sentence, or several sentences in order within a speech up until the “bucket” fills up to the threshold of <img src="https://latex.codecogs.com/png.latex?30"> seconds.</p>
<p>We remove the first sentence of each speech, as transcriptions tend to add a “Fru talman!” or “Herr Talman” here as a matter of formality, regardless of whether this was uttered by the speaker or not.</p>
</section>
<section id="dataset-card" class="level3">
<h3 class="anchored" data-anchor-id="dataset-card">Dataset card</h3>
<p>RixVox has <a href="https://huggingface.co/datasets/KBLab/rixvox">a dataset card</a> on Huggingface, where you can find more details about the dataset, its features, and how to download and use it. You can also preview the first 100 observations of the train, validation and test sets in <a href="https://huggingface.co/datasets/KBLab/rixvox/viewer/default/train">the dataset viewer</a>.</p>
</section>
</section>
<section id="acknowledgements" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="acknowledgements">Acknowledgements</h2>
<p>Part of this development work was carried out within the frame of the infrastructural project <a href="https://www.huminfra.se/">HUMINFRA</a>.</p>

<div class="no-row-height column-margin column-container"><div class="">
<p><img src="https://kb-labb.github.io/posts/2023-02-15-finding-speeches-in-the-riksdags-debates/huminfra.svg" class="img-fluid" style="width:35.0%"></p>
</div></div></section>


<div id="quarto-appendix" class="default"><section id="code" class="level2 appendix"><h2 class="anchored quarto-appendix-heading">Code</h2><div class="quarto-appendix-contents">

<p>The code for reproducing results in this article can be found on https://github.com/kb-labb/riksdagen_anforanden.</p>



</div></section><section class="quarto-appendix-contents" id="quarto-bibliography"><h2 class="anchored quarto-appendix-heading">References</h2><div id="refs" class="references csl-bib-body hanging-indent" data-entry-spacing="0">
<div id="ref-Biczysko1674281" class="csl-entry">
Biczysko, Klaudia. 2022. <span>“Automatic Annotation of Speech: Exploring Boundaries Within Forced Alignment for Swedish and Norwegian.”</span> Master’s thesis, Uppsala University, Department of Linguistics; Philology; Uppsala University, Department of Linguistics; Philology.
</div>
<div id="ref-wav2vec2" class="csl-entry">
Malmsten, Martin, Chris Haffenden, and Love Börjeson. 2022. <span>“Hearing Voices at the National Library – a Speech Corpus and Acoustic Model for the Swedish Language.”</span> <a href="https://arxiv.org/abs/2205.03026">https://arxiv.org/abs/2205.03026</a>.
</div>
<div id="ref-rekathati2023finding" class="csl-entry">
Rekathati, Faton. 2023. <span>“The KBLab Blog: Finding Speeches in the Riksdag’s Debates.”</span> <a href="https://kb-labb.github.io/posts/2023-02-15-finding-speeches-in-the-riksdags-debates/">https://kb-labb.github.io/posts/2023-02-15-finding-speeches-in-the-riksdags-debates/</a>.
</div>
</div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{rekathati2023,
  author = {Rekathati, Faton},
  title = {RixVox: {A} {Swedish} {Speech} {Corpus} with 5500 {Hours} of
    {Speech} from {Parliamentary} {Debates}},
  date = {2023-03-09},
  url = {https://kb-labb.github.io/posts/2023-03-09-rixvox-a-swedish-speech-corpus/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-rekathati2023" class="csl-entry quarto-appendix-citeas">
Rekathati, Faton. 2023. <span>“RixVox: A Swedish Speech Corpus with 5500
Hours of Speech from Parliamentary Debates.”</span> March 9, 2023. <a href="https://kb-labb.github.io/posts/2023-03-09-rixvox-a-swedish-speech-corpus/">https://kb-labb.github.io/posts/2023-03-09-rixvox-a-swedish-speech-corpus/</a>.
</div></div></section></div> ]]></description>
  <guid>https://kb-labb.github.io/posts/2023-03-09-rixvox-a-swedish-speech-corpus/</guid>
  <pubDate>Wed, 08 Mar 2023 23:00:00 GMT</pubDate>
  <media:content url="https://kb-labb.github.io/posts/2023-03-09-rixvox-a-swedish-speech-corpus/rixvox_bg.png" medium="image" type="image/png" height="144" width="144"/>
</item>
<item>
  <title>Finding Speeches in the Riksdag’s Debates</title>
  <dc:creator>Faton Rekathati</dc:creator>
  <link>https://kb-labb.github.io/posts/2023-02-15-finding-speeches-in-the-riksdags-debates/</link>
  <description><![CDATA[ 





<div class="cell">
<div class="cell-output-display">
<script>var data_example = [{"dokid":"H9C120220512fs","start":94,"end":157,"start_adjusted":97.5,"end_adjusted":154,"speaker":"Ann-Sofie Alm","party":"M","speaker_audio_meta":"Ann-Sofie Alm (M)","anftext_short":"Fru talman! Njutningsäktenskap låter härligt. Men Uppdrag granskning visade att det är religiöst sanktionerat koppleri. Det är sexhandel, det är...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":157,"end":208,"start_adjusted":163.2,"end_adjusted":207.4,"speaker":"Matilda Ernkrans","party":"S","speaker_audio_meta":"Statsrådet Matilda Ernkrans (S)","anftext_short":"Fru talman! Jag och regeringen delar absolut uppfattningen att det här är helt oanständigt. Det är dessutom förbjudet i lag...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":208,"end":244,"start_adjusted":211.1,"end_adjusted":243.5,"speaker":"Ann-Sofie Alm","party":"M","speaker_audio_meta":"Ann-Sofie Alm (M)","anftext_short":"Fru talman! Att män anser sig ha rätten att äga kvinnor samtidigt som jämställdhetens fana vajar högt utan att någonting...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":244,"end":276,"start_adjusted":246.5,"end_adjusted":273.7,"speaker":"Matilda Ernkrans","party":"S","speaker_audio_meta":"Statsrådet Matilda Ernkrans (S)","anftext_short":"Fru talman! Kvinnors och flickors rättigheter och jämställdhet står högt på regeringens agenda. Detta är ett socialdemokratiskt signum sedan många...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":276,"end":337,"start_adjusted":280.2,"end_adjusted":332.2,"speaker":"Henrik Vinge","party":"SD","speaker_audio_meta":"Henrik Vinge (SD)","anftext_short":"Fru talman! De som har jobbat och slitit ett helt yrkesliv ska få en värdig ålderdom och en bra pension;...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":337,"end":378,"start_adjusted":340.9,"end_adjusted":377.1,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Socialminister Lena Hallengren (S)","anftext_short":"Fru talman! Det är välkommet att högerpartierna äntligen är med och diskuterar och till och med lägger fram förslag som...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":378,"end":409,"start_adjusted":378.4,"end_adjusted":407.9,"speaker":"Henrik Vinge","party":"SD","speaker_audio_meta":"Henrik Vinge (SD)","anftext_short":"Fru talman! Vi har länge sträckt ut handen till Socialdemokraterna och sagt: Ska vi inte lösa pensionsfrågan? Men Socialdemokraterna har...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":409,"end":442,"start_adjusted":411.1,"end_adjusted":440.3,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Socialminister Lena Hallengren (S)","anftext_short":"Fru talman! Jag tackar för möjligheten att förtydliga detta. Det finns ju en pensionsgrupp. Det är otroligt viktigt att den...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":442,"end":515,"start_adjusted":447.4,"end_adjusted":513.7,"speaker":"Niels Paarup-Petersen","party":"C","speaker_audio_meta":"Niels Paarup-Petersen (C)","anftext_short":"Fru talman! Kalla fakta har avslöjat hur den kommunala grundskolan Sorgenfriskolan i Malmö under minst tio års tid har avskilt...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":515,"end":579,"start_adjusted":517.7,"end_adjusted":577.9,"speaker":"Anna Ekström","party":"S","speaker_audio_meta":"Utbildningsminister Anna Ekström (S)","anftext_short":"Fru talman! Tack, Niels Paarup-Petersen, för frågan! Det är ingen tvekan om att barn och elever som går i svensk...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":579,"end":618,"start_adjusted":581.5,"end_adjusted":617.6,"speaker":"Niels Paarup-Petersen","party":"C","speaker_audio_meta":"Niels Paarup-Petersen (C)","anftext_short":"Fru talman! Det är absolut ett huvudmannaansvar. Det som är lite anmärkningsvärt här är att ministern själv har lyft just...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":618,"end":655,"start_adjusted":621,"end_adjusted":653.6,"speaker":"Anna Ekström","party":"S","speaker_audio_meta":"Utbildningsminister Anna Ekström (S)","anftext_short":"Fru talman! Skälet till att jag och många andra har lyft Malmö stad, Malmö kommun, är det mycket stora ansvar...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":655,"end":716,"start_adjusted":659.3,"end_adjusted":713.9,"speaker":"Ciczie Weidby","party":"V","speaker_audio_meta":"Ciczie Weidby (V)","anftext_short":"Fru talman! Enligt en rapport om äldres arbetsmiljö som Arbetsmiljöverket släppte för några veckor sedan uppger allt fler att de...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":716,"end":781,"start_adjusted":718.9,"end_adjusted":780.7,"speaker":"Johan Danielsson","party":"S","speaker_audio_meta":"Statsrådet Johan Danielsson (S)","anftext_short":"Fru talman! Jag tackar Ciczie Weidby för frågan. Det är klart att en central del i regeringens arbetsmiljöarbete är att...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":782,"end":815,"start_adjusted":782.8,"end_adjusted":814.2,"speaker":"Ciczie Weidby","party":"V","speaker_audio_meta":"Ciczie Weidby (V)","anftext_short":"Frågestund Fru talman! Jag har även tidigare hört statsrådet prata om de fina ambitionerna, de arbetsmiljöstrategier som man skriver och...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":815,"end":855,"start_adjusted":818.4,"end_adjusted":853.8,"speaker":"Johan Danielsson","party":"S","speaker_audio_meta":"Statsrådet Johan Danielsson (S)","anftext_short":"Fru talman! Regeringen har konsekvent under åtta års tid tillfört mer resurser till Arbetsmiljöverket för att stärka arbetsmiljöarbetet. Vi har...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":855,"end":919,"start_adjusted":859.5,"end_adjusted":917.4,"speaker":"Pia Steensland","party":"KD","speaker_audio_meta":"Pia Steensland (KD)","anftext_short":"Fru talman! Ungefär 145 000 människor står i dag i vårdkö längre än vad vårdgarantin tillåter. För att kapa vårdköerna har...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":919,"end":968,"start_adjusted":921.5,"end_adjusted":966.9,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Socialminister Lena Hallengren (S)","anftext_short":"Fru talman! Jag tackar Pia Steensland för frågan. Nej, vård ska ges efter behov och inte på någon annan grund....","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":968,"end":1000,"start_adjusted":970.3,"end_adjusted":999.3,"speaker":"Pia Steensland","party":"KD","speaker_audio_meta":"Pia Steensland (KD)","anftext_short":"Fru talman! Det är ju det här som riksdagen har begärt: att regeringen ska införa den nationella vårdförmedlingen och att...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":1000,"end":1030,"start_adjusted":1002.5,"end_adjusted":1028.2,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Socialminister Lena Hallengren (S)","anftext_short":"Fru talman! Det finns ingenting som tyder på att det skulle vara så som Pia Steensland säger, nämligen att en...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":1030,"end":1090,"start_adjusted":1034.4,"end_adjusted":1087.7,"speaker":"Barbro Westerholm","party":"L","speaker_audio_meta":"Barbro Westerholm (L)","anftext_short":"Fru talman! Min fråga riktar sig till socialminister Lena Hallengren. När kommer frågan om finansiering av så kallade särläkemedel att...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":1090,"end":1160,"start_adjusted":1092.3,"end_adjusted":1159,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Socialminister Lena Hallengren (S)","anftext_short":"Fru talman! Jag tackar Barbro Westerholm för en otroligt viktig fråga. Det är också en väldigt komplicerad fråga. Det vet...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":1160,"end":1188,"start_adjusted":1160.3,"end_adjusted":1185.8,"speaker":"Barbro Westerholm","party":"L","speaker_audio_meta":"Barbro Westerholm (L)","anftext_short":"Fru talman! Tandvårds- och läkemedelsförmånsverket har idéer om hur detta kan lösas, men man behöver ett regeringsuppdrag. Vi har råd...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":1188,"end":1224,"start_adjusted":1190.3,"end_adjusted":1222.5,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Socialminister Lena Hallengren (S)","anftext_short":"Fru talman! Jag instämmer i att vi ska ha en jämlik vård och att vi ska ha vård efter behov,...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":1224,"end":1286,"start_adjusted":1228.1,"end_adjusted":1284.3,"speaker":"Emma Hult","party":"MP","speaker_audio_meta":"Emma Hult (MP)","anftext_short":"Fru talman! Sverige behöver en ny könstillhörighetslag och det nu. En ny, modern könstillhörighetslag är av livsavgörande betydelse för de...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":1316,"end":1344,"start_adjusted":1317.1,"end_adjusted":1342.9,"speaker":"Emma Hult","party":"MP","speaker_audio_meta":"Emma Hult (MP)","anftext_short":"Fru talman! Jag tackar statsrådet för svaret. Statsrådet får gärna precisera \"förslag till ny lagstiftning\". Handlar det om en ny...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":1358,"end":1412,"start_adjusted":1367.2,"end_adjusted":1410.6,"speaker":"Sofia Amloh","party":"S","speaker_audio_meta":"Sofia Amloh (S)","anftext_short":"Fru talman! Min fråga går till statsrådet Hallengren. Både Myndigheten för vård- och omsorgsanalys och Dagens Nyheter har visat att...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":1412,"end":1468,"start_adjusted":1415,"end_adjusted":1466.1,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Socialminister Lena Hallengren (S)","anftext_short":"Fru talman! Tack, Sofia Amloh, för frågan! Som redan sagts här några gånger tidigare ska vård ges efter behov och...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":1490,"end":1525,"start_adjusted":1492.1,"end_adjusted":1523.5,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Socialminister Lena Hallengren (S)","anftext_short":"Fru talman! Jag tänker att sista ordet definitivt inte är sagt utan att vi hela tiden måste vara beredda att...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":1525,"end":1593,"start_adjusted":1529.2,"end_adjusted":1591.8,"speaker":"Viktor Wärnick","party":"M","speaker_audio_meta":"Viktor Wärnick (M)","anftext_short":"Fru talman! Vi moderater genomför just nu ett bostadspolitiskt projekt som vi har valt att kalla Arkitekturresan. Vi besöker olika...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":1593,"end":1664,"start_adjusted":1598.9,"end_adjusted":1662,"speaker":"Johan Danielsson","party":"S","speaker_audio_meta":"Statsrådet Johan Danielsson (S)","anftext_short":"Fru talman! Tack, Viktor Wärnick, för frågan! Att skapa hållbara och goda livsmiljöer är givetvis extremt viktigt för människors välbefinnande...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":1664,"end":1734,"start_adjusted":1668.3,"end_adjusted":1733.1,"speaker":"Linda Lindberg","party":"SD","speaker_audio_meta":"Linda Lindberg (SD)","anftext_short":"Fru talman! Förlossningsvården är i dag i stor kris. Från norr till söder rapporterar man om personalbrist och dålig arbetsmiljö....","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":1734,"end":1797,"start_adjusted":1736.4,"end_adjusted":1796.1,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Socialminister Lena Hallengren (S)","anftext_short":"Fru talman! Jag tackar Linda Lindberg för en innehållsrik fråga. Det finns så många dimensioner i den att jag nästan...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":1797,"end":1852,"start_adjusted":1803.5,"end_adjusted":1851.2,"speaker":"Rickard Nordin","party":"C","speaker_audio_meta":"Rickard Nordin (C)","anftext_short":"Fru talman! Befintliga projekt inom Europeiska regionala utvecklingsfonden har fått en möjlighet till förlängning tack vare olika åtgärder och en...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":1852,"end":1891,"start_adjusted":1855.2,"end_adjusted":1889.5,"speaker":"Matilda Ernkrans","party":"S","speaker_audio_meta":"Statsrådet Matilda Ernkrans (S)","anftext_short":"Fru talman! Tack, Rickard Nordin, för frågan! Det är kanske en fråga som är på en väldigt detaljerad nivå när...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":1891,"end":1959,"start_adjusted":1895.9,"end_adjusted":1956.4,"speaker":"Yasmine Posio","party":"V","speaker_audio_meta":"Yasmine Posio (V)","anftext_short":"Fru talman! Världen befinner sig i ett exceptionellt läge. Redan före pandemin såg vi hur rika blev rikare samtidigt som...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":1959,"end":2015,"start_adjusted":1961.9,"end_adjusted":2014.2,"speaker":"Matilda Ernkrans","party":"S","speaker_audio_meta":"Statsrådet Matilda Ernkrans (S)","anftext_short":"Fru talman! Sverige är mycket bra på internationell solidaritet. Vi är ett av de länder i världen - topp tre...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":2015,"end":2075,"start_adjusted":2021.3,"end_adjusted":2073.4,"speaker":"Gudrun Brunegård","party":"KD","speaker_audio_meta":"Gudrun Brunegård (KD)","anftext_short":"Fru talman! Regeringen har satt ett tak för utbetalningar till internationellt bistånd på 9,2 miljarder under innevarande år. Nu kräver...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":2075,"end":2139,"start_adjusted":2078.5,"end_adjusted":2137.8,"speaker":"Matilda Ernkrans","party":"S","speaker_audio_meta":"Statsrådet Matilda Ernkrans (S)","anftext_short":"Fru talman! Sverige är ett land man kan lita på, åtminstone så länge vi har Magdalena Andersson som statsminister och...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":2140,"end":2208,"start_adjusted":2142.9,"end_adjusted":2206.9,"speaker":"Roger Haddad","party":"L","speaker_audio_meta":"Roger Haddad (L)","anftext_short":"Fru talman! Svensk skola har i dag många utmaningar. Vi har bristande kvalitet på många håll, och vi har bristande...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":2208,"end":2276,"start_adjusted":2210.1,"end_adjusted":2273.9,"speaker":"Anna Ekström","party":"S","speaker_audio_meta":"Utbildningsminister Anna Ekström (S)","anftext_short":"Fru talman! Detta är en fråga som hör till statsrådet Lina Axelsson Kihlbloms ansvarsområde, men jag kan rent övergripande säga...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":2276,"end":2337,"start_adjusted":2281.1,"end_adjusted":2335.2,"speaker":"Margareta Fransson","party":"MP","speaker_audio_meta":"Margareta Fransson (MP)","anftext_short":"Fru talman! I vårt välfärdssamhälle har vi kunnat ta del av skrämmande uppgifter om 170 lex Maria-anmälningar de senaste tre...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":2337,"end":2402,"start_adjusted":2339.4,"end_adjusted":2399.9,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Socialminister Lena Hallengren (S)","anftext_short":"Fru talman! Tack, Margareta Fransson, för frågan! Det är uppenbart att den engagerar inte bara i denna kammare utan i...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":2402,"end":2467,"start_adjusted":2406.2,"end_adjusted":2465.9,"speaker":"Anna Johansson","party":"S","speaker_audio_meta":"Anna Johansson (S)","anftext_short":"Fru talman! Min fråga går till statsrådet Johan Danielsson. Arbetslivskriminaliteten är ett stort problem i Sverige. Människor utnyttjas, och samhällets...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":2467,"end":2538,"start_adjusted":2469.3,"end_adjusted":2536.6,"speaker":"Johan Danielsson","party":"S","speaker_audio_meta":"Statsrådet Johan Danielsson (S)","anftext_short":"Fru talman! Jag tackar Anna Johansson för frågan. Sekretesshinder är mycket riktigt en ständigt återkommande fråga kopplad till det myndighetsgemensamma...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":2538,"end":2614,"start_adjusted":2543.2,"end_adjusted":2610.4,"speaker":"Amineh Kakabaveh","party":"-","speaker_audio_meta":"Amineh Kakabaveh (-)","anftext_short":"Fru talman! Vi är många som är riktigt upprörda över avslöjandena i Uppdrag granskning om att shiamuslimska imamer i Sverige...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":2614,"end":2661,"start_adjusted":2617.3,"end_adjusted":2658.5,"speaker":"Matilda Ernkrans","party":"S","speaker_audio_meta":"Statsrådet Matilda Ernkrans (S)","anftext_short":"Fru talman! Jag tackar Amineh Kakabaveh för frågan. Ditt långa engagemang i dessa frågor är väl känt. Och jag vill...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":2661,"end":2732,"start_adjusted":2666,"end_adjusted":2730.4,"speaker":"Camilla Waltersson Grönvall","party":"M","speaker_audio_meta":"Camilla Waltersson Grönvall (M)","anftext_short":"Fru talman! För en tid sedan överlämnade regeringen propositionen om lex Lilla hjärtat till riksdagen. Där saknade vi framför allt...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":2732,"end":2795,"start_adjusted":2734.5,"end_adjusted":2792.8,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Socialminister Lena Hallengren (S)","anftext_short":"Fru talman! Jag tackar Camilla Waltersson Grönvall för en fråga i det spann som vi har diskuterat många gånger -...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":2795,"end":2863,"start_adjusted":2798.4,"end_adjusted":2861.5,"speaker":"Markus Wiechel","party":"SD","speaker_audio_meta":"Markus Wiechel (SD)","anftext_short":"Fru talman! Ända sedan Rysslands hänsynslösa invasion av Ukraina har budskapet från president Zelenskyj varit tydligt. De behöver mer läkemedel,...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":2863,"end":2916,"start_adjusted":2864.3,"end_adjusted":2913.3,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Socialminister Lena Hallengren (S)","anftext_short":"Fru talman! Jag tackar Markus Wiechel för frågan. Jag vet faktiskt inte varifrån Markus Wiechel hämtar sin information. Men den...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":2916,"end":2984,"start_adjusted":2919.7,"end_adjusted":2982.4,"speaker":"Magnus Ek","party":"C","speaker_audio_meta":"Magnus Ek (C)","anftext_short":"Fru talman! Även min fråga rör de historiskt stora avräkningarna från det svenska biståndet. Jag tror inte att det är...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":2984,"end":3055,"start_adjusted":2987.2,"end_adjusted":3053.7,"speaker":"Matilda Ernkrans","party":"S","speaker_audio_meta":"Statsrådet Matilda Ernkrans (S)","anftext_short":"Fru talman! Det första jag ska säga är att vi har ett krig i Europa och en stor flyktingkris. Det...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":3055,"end":3122,"start_adjusted":3059.3,"end_adjusted":3120.8,"speaker":"Magnus Jacobsson","party":"KD","speaker_audio_meta":"Magnus Jacobsson (KD)","anftext_short":"Fru talman! Min fråga är i dag generell, och jag vänder mig därför till Matilda Ernkrans. Sverige är ett fantastiskt...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":3122,"end":3167,"start_adjusted":3124.4,"end_adjusted":3165.4,"speaker":"Matilda Ernkrans","party":"S","speaker_audio_meta":"Statsrådet Matilda Ernkrans (S)","anftext_short":"Fru talman! Sveriges regering har tre prioriteringar för att vi ska kunna bygga det här landet starkare. Vi ska se...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":3167,"end":3219,"start_adjusted":3172.3,"end_adjusted":3217.5,"speaker":"Johanna Haraldsson","party":"S","speaker_audio_meta":"Johanna Haraldsson (S)","anftext_short":"Fru talman! Kompetensförsörjningen är en av de största utmaningarna för svenska företag, för svensk konkurrenskraft och i förlängningen för svensk...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":3219,"end":3289,"start_adjusted":3222.3,"end_adjusted":3286.8,"speaker":"Anna Ekström","party":"S","speaker_audio_meta":"Utbildningsminister Anna Ekström (S)","anftext_short":"Fru talman! Jag tackar Johanna Haraldsson för en väldigt viktig fråga. Runt om i vårt land växer det fram nya,...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":3289,"end":3351,"start_adjusted":3293.2,"end_adjusted":3349.7,"speaker":"Johan Hultberg","party":"M","speaker_audio_meta":"Johan Hultberg (M)","anftext_short":"Fru talman! I Sverige är aborträtten stark, men samtidigt har den svenska abortlagstiftningen blivit föråldrad. När abortlagen trädde i kraft...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":3351,"end":3422,"start_adjusted":3353.7,"end_adjusted":3420.3,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Socialminister Lena Hallengren (S)","anftext_short":"Fru talman! Jag tänker att hanteringen av just den frågan väl visar en del om hur riksdagen nuförtiden arbetar, nämligen...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":3489,"end":3560,"start_adjusted":3493,"end_adjusted":3557.6,"speaker":"Johan Danielsson","party":"S","speaker_audio_meta":"Statsrådet Johan Danielsson (S)","anftext_short":"Fru talman! Jag tackar Magnus Persson för frågan. I regeringen kommer vi inte att vara nöjda förrän vi har uppnått...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":3560,"end":3607,"start_adjusted":3565.4,"end_adjusted":3605.5,"speaker":"Diana Laitinen Carlsson","party":"S","speaker_audio_meta":"Diana Laitinen Carlsson (S)","anftext_short":"Fru talman! Kriget i Ukraina visar med otvetydig klarhet på vikten av ett robust, högt och tryggt bistånd, ett bistånd...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":3607,"end":3678,"start_adjusted":3609.3,"end_adjusted":3675.3,"speaker":"Matilda Ernkrans","party":"S","speaker_audio_meta":"Statsrådet Matilda Ernkrans (S)","anftext_short":"Fru talman! Jag tackar för frågan. Sverige har tre fokus när det gäller Ukraina: Vi ska stötta Ukraina, vi ska...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":3678,"end":3727,"start_adjusted":3682.4,"end_adjusted":3725.3,"speaker":"David Josefsson","party":"M","speaker_audio_meta":"David Josefsson (M)","anftext_short":"Fru talman! Nio av tio svenskar bor i dag i en kommun med bostadsbrist. Priserna på småhus och bostadsrätter har...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":3727,"end":3798,"start_adjusted":3731.2,"end_adjusted":3796.7,"speaker":"Johan Danielsson","party":"S","speaker_audio_meta":"Statsrådet Johan Danielsson (S)","anftext_short":"Fru talman! Tack för frågan, David Josefsson! Det är bra att vi delar uppfattningen att den utbredda bostadsbristen i Sverige...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":3798,"end":3862,"start_adjusted":3804.1,"end_adjusted":3860.2,"speaker":"Mikael Eskilandersson","party":"SD","speaker_audio_meta":"Mikael Eskilandersson (SD)","anftext_short":"Fru talman! Allt börjar med en bra bostad. Sverigedemokraterna tänker inte stå och se på medan tryggheten försvinner och allt...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":3862,"end":3928,"start_adjusted":3864.6,"end_adjusted":3925.9,"speaker":"Johan Danielsson","party":"S","speaker_audio_meta":"Statsrådet Johan Danielsson (S)","anftext_short":"Fru talman! Tack till Mikael Eskilandersson för frågan! Det är mycket riktigt en prioriterad fråga för statsministern och för den...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":3928,"end":3998,"start_adjusted":3932.5,"end_adjusted":3997,"speaker":"Marie-Louise Hänel Sandström","party":"M","speaker_audio_meta":"Marie-Louise Hänel Sandström (M)","anftext_short":"Fru talman! Min fråga går till utbildningsminister Anna Ekström. I förra veckan hade vi en forskningsdebatt här i kammaren. Moderaterna...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":3998,"end":4068,"start_adjusted":4000.4,"end_adjusted":4065.9,"speaker":"Anna Ekström","party":"S","speaker_audio_meta":"Utbildningsminister Anna Ekström (S)","anftext_short":"Fru talman! Jag vill börja med att hålla med Marie-Louise Hänel Sandström om att forskning är mycket viktigt för att...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":4068,"end":4133,"start_adjusted":4072.6,"end_adjusted":4130.2,"speaker":"Ann-Christine From Utterstedt","party":"SD","speaker_audio_meta":"Ann-Christine From Utterstedt (SD)","anftext_short":"Fru talman! Sveriges styrande politiker har under lång tid bedrivit en fullständigt oansvarig migrationspolitik, vilket lett till att vårt land...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":4133,"end":4200,"start_adjusted":4136.1,"end_adjusted":4198.1,"speaker":"Matilda Ernkrans","party":"S","speaker_audio_meta":"Statsrådet Matilda Ernkrans (S)","anftext_short":"Fru talman! Ja, det här är Sverigedemokraterna, som Moderaterna, Kristdemokraterna och Liberalerna tänker sig att regera Sverige tillsammans med. Jag...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":4200,"end":4263,"start_adjusted":4203.1,"end_adjusted":4261.4,"speaker":"Åsa Coenraads","party":"M","speaker_audio_meta":"Åsa Coenraads (M)","anftext_short":"Fru talman! I dag läggs 95 procent av alla våldtäktsanmälningar ned. Samtidigt varnar polisen om att de inte hinner utreda...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"},{"dokid":"H9C120220512fs","start":4263,"end":4314,"start_adjusted":4266,"end_adjusted":4312.6,"speaker":"Matilda Ernkrans","party":"S","speaker_audio_meta":"Statsrådet Matilda Ernkrans (S)","anftext_short":"Fru talman! Vi har skärpt straffen. Vi har öppnat två nya polisutbildningar. Vi utbildar fler poliser än någonsin. Vi har...","debateurl":"/sv/webb-tv/video/fragestund/fragestund_H9C120220512fs"}];</script>
</div>
</div>
<p>The Riksdag is Sweden’s legislature. The 349 members of the Riksdag regularly gather to debate in <a href="https://www.riksdagen.se/en/how-the-riksdag-works/the-work-of-the-riksdag/debates-and-decisions-in-the-chamber/">the Chamber</a> of the <a href="https://en.wikipedia.org/wiki/Parliament_House,_Stockholm">Parliament House</a>. These debates are recorded and published to the Riksdag’s <a href="https://www.riksdagen.se/sv/webb-tv/">Web TV</a>. For the past 20 years the Riksdag’s media recordings have been enriched with further metadata, including tags for the start and the duration of each speech, along with their corresponding transcripts. Speaker lists are added to each debate, allowing viewers to navigate and jump between speeches easier <sup>1</sup>. This metadata also allows linking members of parliament and ministers with the debates they have participated in.</p>

<div class="no-row-height column-margin column-container"><div class="">
<p><a href="riksmotets-oppnande-2022_kammaren2.jpg" class="lightbox" data-gallery="quarto-lightbox-gallery-1"><img src="https://kb-labb.github.io/posts/2023-02-15-finding-speeches-in-the-riksdags-debates/riksmotets-oppnande-2022_kammaren2.jpg" class="img-fluid" width="360"></a></p>
<p>The Chamber.</p>
<p><em>Photo: Melker Dahlstrand /The Swedish Parliament</em></p>
</div></div><p>Happening upon these recordings and seeing them linked to such rich metadata, we were curious to learn if the Riksdag had planned to make them available through its <a href="https://data.riksdagen.se/">open data platform</a> and APIs. We e-mailed them to inquire whether an audio/media file API was in the plans, to which they responded that such an API in fact already did and does exist, although they had yet to settle on a good way of communicating this service to the public.</p>
<p>Part of our work here at KBLab involves training Swedish <em>Automatic Speech Recognition</em> (ASR) models capable of transcribing speech to text. We release these models freely and openly, see for example our <a href="https://huggingface.co/KBLab/wav2vec2-large-voxrex-swedish">wav2vec2-large-voxrex-swedish</a> model <span class="citation" data-cites="wav2vec2">(Malmsten, Haffenden, and Börjeson 2022)</span>. Here, audio datasets with annotated transcriptions play an especially important part for quality. Unfortunately such datasets are hard to come by for Swedish. The combined total of current available datasets with annotated transcriptions number somewhere in the <em>hundreds</em> of hours.</p>
<p>Decades of debates from the Riksdag present a golden opportunity to increase this quantity by a factor of tenfold or more. The wealth of dialects and accents, along with the breadth of metadata covering electoral districts, birth year and the gender of members of parliament may serve to not only improve ASR systems, but also to enable research on the weaknesses and biases of both current and future models.</p>
<p>However, in order to get the speeches and transcripts properly aligned so they can be brought to a suitable and workable format, we first need to ensure the region of audio within a debate representing a speech actually matches the written transcript. Additionally no other speakers should ideally be present in this window.</p>
<section id="the-importance-of-metadata" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="the-importance-of-metadata">The importance of metadata</h2>
<p>When assessing the quality of the available metadata from debates, we found most of the material from 2012 and forward was generally accurate and of high quality. However, the required degree of precision of metadata always depends on one’s use case. And in our case it was important that only one speaker – the one making the speech – be present in the indicated window. With this in mind, it soon became evident that a certain level of adjustment of the existing metadata was required. Illustrating this with an example below, we see that the Riksdag’s metadata (<strong>right video</strong>) tends to include parts where the speaker of the house talks. The <strong>left video</strong> shows the automatically generated metadata from the method we developed for locating and segmenting speeches. We employed <a href="https://en.wikipedia.org/wiki/Speaker_diarisation">speaker diarization</a> to more precisely pinpoint when a speaker starts and stops speaking.</p>

<div class="no-row-height column-margin column-container"><div class="">
<p>Both videos below will update at the same time when pressing “Next debate”/“Previous debate”. The speeches are set to begin and end playing at the “start” and “end” of both metadata sources. Play them one after another to see how they compare in terms of speech segmentation.</p>
</div></div><div class="column-page">
<div class="videobox-container">
  <div class="videobox">
    <iframe id="videokb" src="https://www.riksdagen.se/sv/webb-tv/video/fragestund/fragestund_h9c120220512fs/embed/?start=97.5&amp;end=154" allowfullscreen="" scrolling="no" title="Frågestund (Frågestund 12 maj 2022)" style="width: 510px; height: 288px; border: 0; margin-bottom: 0em;"></iframe>
    
    <div class="metabox">
      <form action="">
        <input type="button" id="prevspeech" value="Previous speech">
        <input type="button" id="nextspeech" value="Next speech">
      </form>
      <h3 class="anchored">Adjusted metadata (KBLab)</h3>
      <p class="beginning"><u>Beginning of official transcript:</u></p>
      <p id="speechtranscript">Fru talman! Njutningsäktenskap låter härligt. Men Uppdrag granskning visade att det är religiöst sanktionerat koppleri. Det är sexhandel, det är...</p>
      <p id="metadatastart"><b>Start:</b> 00:01:37.5  <b>End:</b> 00:02:34.0</p>
      <p id="speaker1"><b>Speaker:</b> Ann-Sofie Alm (M)</p>
      <p><small><i>Source: Sveriges riksdag.</i></small></p>
    </div>
  </div>

  <div class="videobox">
    <iframe id="videoriks" src="https://www.riksdagen.se/sv/webb-tv/video/fragestund/fragestund_h9c120220512fs/embed/?start=94&amp;end=157" allowfullscreen="" scrolling="no" title="Frågestund (Frågestund 12 maj 2022)" style="width: 510px; height: 288px; border: 0; margin-bottom: 0em;"></iframe>
    <div class="metabox">
      <form action="">
        <input type="button" id="prevspeech2" value="Previous speech">
        <input type="button" id="nextspeech2" value="Next speech">
      </form>
      <h3 class="anchored">The Riksdag's metadata</h3>
      <p class="beginning"><u>Beginning of official transcript:</u></p>
      <p id="speechtranscript2">Fru talman! Njutningsäktenskap låter härligt. Men Uppdrag granskning visade att det är religiöst sanktionerat koppleri. Det är sexhandel, det är...</p>
      <p id="metadatastart2"><b>Start:</b> 00:01:34.0  <b>End:</b> 00:02:37.0</p>
      <p id="speaker2"><b>Speaker:</b> Ann-Sofie Alm (M)</p>
      <p><small><i>Source: Sveriges riksdag.</i></small></p>
    </div>
  </div>
</div>
</div>
<p>Modern Riksdag metadata, such as the debate above, serve as a good benchmark against which we can evaluate our fully automated method. Should our method – using only official transcripts and audio – be able to roughly match the segmentation quality of these more recent debates, we can be reasonably certain it can also fare well when applied on older materials.</p>
<p>Below we display speeches from 10 sampled debates from before 2012-01-01 – a period where metadata quality tends to be shakier. The first and the second debate, “Regeringens skärpning av migrationspolitiken” and “Kvalitet i förskolan m.m.”, contain large errors in the Riksdag’s metadata when it comes to start and end times. It appears the metadata in these debates have shifted to be off by an entire speech. In addition to the above, we found mismatches between the indicated names of the speakers in text form and in the internal id-system that the Riksdag use to identify members of parliament. The more accurate field is <code>intresent_id</code> which lists the id number of the speaker, whereas the <code>text</code> field which lists the name in textual form at times can be misleading.</p>
<p>Likely this is either an off by one error during data entry, a joining of disparate datasets gone wrong, or some post-processing mistake. Since we found the <code>intressent_id</code> field to be reliable, we used the id’s to fetch the names and information of parliament members from a separate data file the Riksdag provides in their open data platform <sup>2</sup>.</p>
<div class="cell">
<div class="cell-output-display">
<script>var data_debate = {"GR10298":[{"start":0,"end":312,"start_adjusted":755.4,"end_adjusted":1001.9,"speaker":"Erik Ullenhag","party":"L","speaker_audio_meta":"Barbro Holmberg (S)","anftext_short":"Herr talman! Tyvärr tvingas jag notera att jag inte fick svar på de frågor jag ställde, men jag kan upprepa...","start_diff":755.5022,"end_diff":689.5397,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/regeringens-skarpning-av-migrationspolitiken_GR10298"},{"start":308,"end":565,"start_adjusted":1004.8,"end_adjusted":1200.1,"speaker":"Barbro Holmberg","party":"S","speaker_audio_meta":"Erik Ullenhag (Fp)","anftext_short":"Herr talman! Erik Ullenhag säger att det viktigaste är att vi får ned handläggningstiderna. Det kan jag verkligen hålla med...","start_diff":696.9484,"end_diff":634.6522,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/regeringens-skarpning-av-migrationspolitiken_GR10298"},{"start":562,"end":763,"start_adjusted":1210.3,"end_adjusted":1340.4,"speaker":"Erik Ullenhag","party":"L","speaker_audio_meta":"Barbro Holmberg (S)","anftext_short":"Herr talman! Det kanske förvånar Barbro Holmberg, men det händer ganska ofta att jag är ute och reser i landet....","start_diff":648.4016,"end_diff":576.9678,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/regeringens-skarpning-av-migrationspolitiken_GR10298"},{"start":758,"end":1017,"start_adjusted":1344.1,"end_adjusted":1471.3,"speaker":"Barbro Holmberg","party":"S","speaker_audio_meta":"Erik Ullenhag (Fp)","anftext_short":"Herr talman! Erik Ullenhag säger att han har rest omkring och diskuterat integrationsfrågor. Jag skulle vilja föreslå att Erik Ullenhag...","start_diff":586.2203,"end_diff":453.8672,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/regeringens-skarpning-av-migrationspolitiken_GR10298"}],"GS01UBU3":[{"start":389,"end":886,"start_adjusted":0.4,"end_adjusted":369.6,"speaker":"Margareta Pålsson","party":"M","speaker_audio_meta":"Christer Nylander (Fp)","anftext_short":"Herr talman! Tänk om jag hade fått skriva en proposition om kvalitet i förskolan. Då skulle jag ha skrivit om...","start_diff":-388.5022,"end_diff":-516.8341,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":883,"end":1270,"start_adjusted":385.9,"end_adjusted":860.2,"speaker":"Christer Nylander","party":"L","speaker_audio_meta":"Inger Davidson (Kd)","anftext_short":"Herr talman! Den proposition som behandlas här beskrivs kortsiktigt med några rader: Fatima leker med lego. Man kan nästan se...","start_diff":-497.0097,"end_diff":-410.1766,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":1259,"end":1814,"start_adjusted":874.6,"end_adjusted":1243.1,"speaker":"Inger Davidson","party":"KD","speaker_audio_meta":"Sofia Larsen (C)","anftext_short":"Herr talman! Debatten i dag handlar om våra barn och hur de ska tas om hand när de är mellan...","start_diff":-384.2928,"end_diff":-571.3166,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":1811,"end":2228,"start_adjusted":1253.7,"end_adjusted":1788.6,"speaker":"Sofia Larsen","party":"C","speaker_audio_meta":"Mikaela Valtersson (Mp)","anftext_short":"Herr talman! Att vara förälder och få ihop vardagspusslet är inte alltid så lätt. Tiden är en knapp resurs för...","start_diff":-557.1622,"end_diff":-439.7647,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":2218,"end":2662,"start_adjusted":1800.3,"end_adjusted":2202,"speaker":"Mikaela Valtersson","party":"MP","speaker_audio_meta":"Louise Malmström (S)","anftext_short":"Herr talman! Med risk för att bli lite tjatig, eftersom det är tredje gången jag är med i en debatt...","start_diff":-417.5809,"end_diff":-460.4116,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":2658,"end":2687,"start_adjusted":2215.9,"end_adjusted":2635.2,"speaker":"Louise Malmström","party":"S","speaker_audio_meta":"Sofia Larsen (C)","anftext_short":"Herr talman! För några veckor sedan blev jag intervjuad av en doktorand vid Columbia University i New York. Hon hade...","start_diff":-441.9666,"end_diff":-52.1628,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":2791,"end":2837,"start_adjusted":2690.8,"end_adjusted":2782.6,"speaker":"Sofia Larsen","party":"C","speaker_audio_meta":"Louise Malmström (S)","anftext_short":"Herr talman! Det var ett intressant svar. Riksdagen har med majoritet beslutat just att vi ska säkerställa och öka mångfalden...","start_diff":-100.0534,"end_diff":-54.7934,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":2834,"end":2882,"start_adjusted":2784.2,"end_adjusted":2816.8,"speaker":"Louise Malmström","party":"S","speaker_audio_meta":"Margareta Pålsson (M)","anftext_short":"Herr talman! Kommunal och enskild förskoleverksamhet har ett gemensamt uppdrag och ska inte arbeta i konkurrens. De ska samverka med...","start_diff":-49.6672,"end_diff":-65.5878,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":2879,"end":2904,"start_adjusted":2825,"end_adjusted":2860.7,"speaker":"Margareta Pålsson","party":"M","speaker_audio_meta":"Louise Malmström (S)","anftext_short":"Herr talman! Jag skulle vilja fråga om personalen. Förskolan är mycket dess personal. Vi har talat om de 6 000...","start_diff":-53.8634,"end_diff":-43.6959,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":2935,"end":3023,"start_adjusted":2894.7,"end_adjusted":2922.8,"speaker":"Louise Malmström","party":"S","speaker_audio_meta":"Inger Davidson (Kd)","anftext_short":"Herr talman! Jag tror inte att man ska fokusera på det. Det bästa sättet att rekrytera personal till förskolan är...","start_diff":-40.2034,"end_diff":-100.6297,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":3019,"end":3072,"start_adjusted":2930.5,"end_adjusted":3002,"speaker":"Inger Davidson","party":"KD","speaker_audio_meta":"Louise Malmström (S)","anftext_short":"Herr talman! Först undrar jag över den långa uppräkningen av förskolans fördelar. De är många, vill jag säga. Men vart...","start_diff":-88.4284,"end_diff":-70.4353,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":3068,"end":3115,"start_adjusted":3004.2,"end_adjusted":3047.1,"speaker":"Louise Malmström","party":"S","speaker_audio_meta":"Inger Davidson (Kd)","anftext_short":"Herr talman! Jag tror att man ska titta på vad det väger över mot. Om det väger över mot lärande...","start_diff":-63.6509,"end_diff":-68.3284,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":3104,"end":3144,"start_adjusted":3048.1,"end_adjusted":3090,"speaker":"Inger Davidson","party":"KD","speaker_audio_meta":"Louise Malmström (S)","anftext_short":"Herr talman! Det är klart att vi kan hjälpa till att stötta upp. Men det vore bra om vi också...","start_diff":-55.7928,"end_diff":-54.4153,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":3140,"end":3219,"start_adjusted":3091.5,"end_adjusted":3120.9,"speaker":"Louise Malmström","party":"S","speaker_audio_meta":"Christer Nylander (Fp)","anftext_short":"Herr talman! Jag tror inte att det faktum att förskolan blir en egen skolform behöver betyda att det måste vara...","start_diff":-48.4072,"end_diff":-98.5341,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":3206,"end":3292,"start_adjusted":3130.4,"end_adjusted":3193.7,"speaker":"Christer Nylander","party":"L","speaker_audio_meta":"Louise Malmström (S)","anftext_short":"Herr talman! Först och främst vill jag säga att det kändes ödmjukt när Louise Malmström tillstod att det inte bara...","start_diff":-75.4766,"end_diff":-98.6678,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":3280,"end":3329,"start_adjusted":3196.4,"end_adjusted":3267.1,"speaker":"Louise Malmström","party":"S","speaker_audio_meta":"Christer Nylander (Fp)","anftext_short":"Herr talman! Först tror jag inte att riksdagen kan gå in och bestämma hur varje enskild barngrupp ska se ut...","start_diff":-83.5291,"end_diff":-62.2616,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":3326,"end":3344,"start_adjusted":3269.1,"end_adjusted":3366.8,"speaker":"Christer Nylander","party":"L","speaker_audio_meta":"Louise Malmström (S)","anftext_short":"Herr talman! Det här är inget stort problem. Men det finns en del områden där det inte är särskilt stora...","start_diff":-56.7641,"end_diff":22.3684,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":3321,"end":3389,"start_adjusted":3367.9,"end_adjusted":3430.1,"speaker":"Louise Malmström","party":"S","speaker_audio_meta":"Christer Nylander (Fp)","anftext_short":"Herr talman! Menar du då, Christer Nylander, att vi i riksdagen ska styra hur stora grupper man ska ha på...","start_diff":47.0222,"end_diff":40.6666,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":3968,"end":4045,"start_adjusted":3455.2,"end_adjusted":3940.3,"speaker":"Lennart Gustavsson","party":"V","speaker_audio_meta":"Christer Nylander (Fp)","anftext_short":"Herr talman! Låt oss först vara överens om att när vi diskuterar förskolan så har alla barnens bästa framför sig....","start_diff":-512.6834,"end_diff":-105.1347,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":4041,"end":4158,"start_adjusted":3949.7,"end_adjusted":4018.7,"speaker":"Christer Nylander","party":"L","speaker_audio_meta":"Lennart Gustavsson (V)","anftext_short":"Herr talman! Jag vill börja med att tacka för de ödmjuka fria funderingar som Lennart Gustavsson hade i sin avslutning....","start_diff":-91.2459,"end_diff":-139.7334,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":4154,"end":4207,"start_adjusted":4020.3,"end_adjusted":4129.3,"speaker":"Lennart Gustavsson","party":"V","speaker_audio_meta":"Christer Nylander (Fp)","anftext_short":"Herr talman! Jag ska börja med att vara ärlig. Jag har inte läst delegationens betänkande. Jag tror att det är...","start_diff":-133.5903,"end_diff":-78.1178,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":4194,"end":4306,"start_adjusted":4132.6,"end_adjusted":4182,"speaker":"Christer Nylander","party":"L","speaker_audio_meta":"Lennart Gustavsson (V)","anftext_short":"Fru talman! Jag har inte heller läst hela betänkandet men åtminstone delar av det. En intressant aspekt i detta är...","start_diff":-61.3041,"end_diff":-124.4003,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":4300,"end":4799,"start_adjusted":4184.6,"end_adjusted":4282.4,"speaker":"Lennart Gustavsson","party":"V","speaker_audio_meta":"Lena Hallengren (S)","anftext_short":"Fru talman! Jag har en tilltro till Jämställdhetsdelegationen som har lämnat ett delbetänkande som jag ska läsa in mig på....","start_diff":-115.2953,"end_diff":-516.9772,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":4790,"end":4820,"start_adjusted":4304.5,"end_adjusted":4769.1,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Sofia Larsen (C)","anftext_short":"Fru talman! Som förskoleminister får jag, liksom säkert många av kammarens ledamöter, brev och mejl från föräldrar. De skriver inte...","start_diff":-485.4153,"end_diff":-51.2684,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":4859,"end":4970,"start_adjusted":4795.6,"end_adjusted":4847.8,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Sofia Larsen (C)","anftext_short":"Fru talman! Jag noterar att Sofia Larsen ställer en mycket kort fråga och inte lägger ut orden så väldigt mycket...","start_diff":-63.2684,"end_diff":-122.5634,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":4967,"end":5079,"start_adjusted":4847.3,"end_adjusted":4947.8,"speaker":"Sofia Larsen","party":"C","speaker_audio_meta":"Lena Hallengren (S)","anftext_short":"Fru talman! Det är synd att förskoleministern inte kan svara på den frågan, för det är ganska intressant att Socialdemokraterna...","start_diff":-119.5634,"end_diff":-131.6128,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":5076,"end":5161,"start_adjusted":4949.3,"end_adjusted":5048.7,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Margareta Pålsson (M)","anftext_short":"Fru talman! Jag tror inte att man behöver vara så rädd. Det är inte vare sig rädsla eller oro som...","start_diff":-126.6047,"end_diff":-112.7172,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":5157,"end":5202,"start_adjusted":5057.8,"end_adjusted":5136.6,"speaker":"Margareta Pålsson","party":"M","speaker_audio_meta":"Lena Hallengren (S)","anftext_short":"Fru talman! Jag tänkte prata lite moral och etik. Det pratas det mycket om i förskolan, och det är viktiga...","start_diff":-99.0984,"end_diff":-65.7816,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":5199,"end":5240,"start_adjusted":5135.4,"end_adjusted":5177,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Margareta Pålsson (M)","anftext_short":"Fru talman! Det är intressant att Margareta Pålsson börjar diskutera moral och etik här i kammaren. Det var nämligen också...","start_diff":-63.5241,"end_diff":-63.3997,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":5227,"end":5275,"start_adjusted":5177.9,"end_adjusted":5216.4,"speaker":"Margareta Pålsson","party":"M","speaker_audio_meta":"Lena Hallengren (S)","anftext_short":"Fru talman! Jag undrar vad 5 miljarder skulle vara till om det inte skulle vara till personal. Det är väl...","start_diff":-49.0497,"end_diff":-59.0472,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":5271,"end":5382,"start_adjusted":5215.9,"end_adjusted":5251.4,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Inger Davidson (Kd)","anftext_short":"Fru talman! Jag kan konstatera att det inte är några riktade pengar som Moderaterna ger till förskolan. Jag kan också...","start_diff":-55.0472,"end_diff":-131.0484,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":5378,"end":5437,"start_adjusted":5258.7,"end_adjusted":5356.8,"speaker":"Inger Davidson","party":"KD","speaker_audio_meta":"Lena Hallengren (S)","anftext_short":"Fru talman! Jag noterade att Lena Hallengren oroar sig för att vi i alliansen ska få ett hårt arbete med...","start_diff":-119.2016,"end_diff":-80.5628,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":5433,"end":5485,"start_adjusted":5356.3,"end_adjusted":5409.9,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Inger Davidson (Kd)","anftext_short":"Fru talman! Jag konstaterar att vi just nu diskuterar förskoleverksamhet och en proposition om kvalitet i förskolan. Jag tycker att...","start_diff":-76.5628,"end_diff":-75.4572,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":5474,"end":5544,"start_adjusted":5410.8,"end_adjusted":5459.1,"speaker":"Inger Davidson","party":"KD","speaker_audio_meta":"Lena Hallengren (S)","anftext_short":"Fru talman! Det kanske är det som är felet. Vi diskuterar inte bara förskolan; vi diskuterar hur våra barn ska...","start_diff":-63.1409,"end_diff":-85.2834,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":5538,"end":5590,"start_adjusted":5459.2,"end_adjusted":5516.1,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Christer Nylander (Fp)","anftext_short":"Fru talman! Det är precis tvärtom. Jag menar att möjligheten att vara hemma med sina barn inte handlar om förskoleverksamhet...","start_diff":-78.6928,"end_diff":-74.3134,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":5586,"end":5606,"start_adjusted":5526.7,"end_adjusted":5562.2,"speaker":"Christer Nylander","party":"L","speaker_audio_meta":"Lena Hallengren (S)","anftext_short":"Fru talman! Lena Hallengren sade att hon längtar efter att få se hur den borgerliga förskolepolitiken blir efter en valseger...","start_diff":-59.2266,"end_diff":-44.2109,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":5602,"end":5621,"start_adjusted":5575.7,"end_adjusted":5619.9,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Christer Nylander (Fp)","anftext_short":"Fru talman! Jag menar att det är varje kommun som bestämmer hur många av vilken yrkeskategori som anställs på vilken...","start_diff":-26.1709,"end_diff":-1.5491,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":5635,"end":5680,"start_adjusted":5647.5,"end_adjusted":5707.8,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Christer Nylander (Fp)","anftext_short":"Fru talman! Jag hade laddat för genusdebatt här, men Christer Nylander byter tema så här i slutet på måndagen. Återigen:...","start_diff":12.5984,"end_diff":27.4372,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":5666,"end":5734,"start_adjusted":5713.6,"end_adjusted":5756.9,"speaker":"Lennart Gustavsson","party":"V","speaker_audio_meta":"Lena Hallengren (S)","anftext_short":"Fru talman! Jag blev lite inspirerad av den tidigare debatten, mest utifrån konstaterandet att 75 % av alla ett- till...","start_diff":47.7316,"end_diff":22.4759,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":5730,"end":5785,"start_adjusted":5757.8,"end_adjusted":5861.6,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Lennart Gustavsson (V)","anftext_short":"Fru talman! Jag trodde först att Lennart Gustavsson skulle säga att 2-3 % är vänsterpartister, men det har vi väl...","start_diff":27.9272,"end_diff":76.1516,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":5773,"end":5890,"start_adjusted":5863.4,"end_adjusted":5946.6,"speaker":"Lennart Gustavsson","party":"V","speaker_audio_meta":"Lena Hallengren (S)","anftext_short":"Fru talman! Procentsatserna för Vänsterpartiet får vi väl se i nästa Temo. Men - för att vara allvarlig - det...","start_diff":90.5309,"end_diff":56.2353,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"},{"start":5878,"end":5976,"start_adjusted":5948.6,"end_adjusted":5986.7,"speaker":"Lena Hallengren","party":"S","speaker_audio_meta":"Lennart Gustavsson (V)","anftext_short":"Fru talman! Jag delar absolut uppfattningen att jämställdhet är en oerhört viktig fråga som löper genom alla politikområden. Jag vill...","start_diff":70.6822,"end_diff":10.3472,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/kvalitet-i-forskolan,-m.m._GS01UBU3"}],"GT10130":[{"start":0,"end":116,"start_adjusted":0.4,"end_adjusted":92.3,"speaker":"Sven-Erik Österberg","party":"S","speaker_audio_meta":"Sven-Erik Österberg (S)","anftext_short":"Herr talman! Tobias Krantz har frågat mig vilka åtgärder jag avser att vidta i syfte att hejda myndighetsraseriet och skapa...","start_diff":0.4978,"end_diff":-24.1072,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/statsforvaltningens-expansion_GT10130"},{"start":105,"end":292,"start_adjusted":96.7,"end_adjusted":267.4,"speaker":"Helena Bargholtz","party":"FP","speaker_audio_meta":"Helena Bargholtz (Fp)","anftext_short":"Herr talman! Att Tobias Krantz och även jag, men framför allt Tobias, tar upp de här frågorna beror som sagt...","start_diff":-8.1797,"end_diff":-24.9784,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/statsforvaltningens-expansion_GT10130"},{"start":281,"end":569,"start_adjusted":275.4,"end_adjusted":544.1,"speaker":"Sven-Erik Österberg","party":"S","speaker_audio_meta":"Sven-Erik Österberg (S)","anftext_short":"Herr talman! Jag skulle vilja säga att den bild som Folkpartiet genom den här interpellationen och även genom det ytterligare...","start_diff":-5.4566,"end_diff":-25.3128,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/statsforvaltningens-expansion_GT10130"},{"start":558,"end":664,"start_adjusted":547.6,"end_adjusted":636.6,"speaker":"Helena Bargholtz","party":"FP","speaker_audio_meta":"Helena Bargholtz (Fp)","anftext_short":"Herr talman! Det låter klokt att statsrådet funderar på dessa myndigheter, och därför är min nyfikna fråga naturligtvis: Vilka myndigheter...","start_diff":-10.3134,"end_diff":-27.8378,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/statsforvaltningens-expansion_GT10130"},{"start":653,"end":911,"start_adjusted":642.4,"end_adjusted":884.3,"speaker":"Sven-Erik Österberg","party":"S","speaker_audio_meta":"Sven-Erik Österberg (S)","anftext_short":"Herr talman! Jag känner till att Folkpartiet inte tycker att regeringen ska inrätta någon ny myndighet utan att samtidigt lägga...","start_diff":-10.5266,"end_diff":-27.0622,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/statsforvaltningens-expansion_GT10130"},{"start":899,"end":1020,"start_adjusted":887.7,"end_adjusted":991.3,"speaker":"Helena Bargholtz","party":"FP","speaker_audio_meta":"Helena Bargholtz (Fp)","anftext_short":"Herr talman! Statsrådet säger att förslaget om en solnedgångsparagraf är naivt. Då ska jag avslöja att det inte är vi...","start_diff":-11.2484,"end_diff":-29.1084,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/statsforvaltningens-expansion_GT10130"},{"start":1010,"end":1156,"start_adjusted":997.3,"end_adjusted":1122.8,"speaker":"Sven-Erik Österberg","party":"S","speaker_audio_meta":"Sven-Erik Österberg (S)","anftext_short":"Herr talman! När det gäller den sista frågan kan jag säga att vi gör det på ett ytterst seriöst sätt....","start_diff":-12.6116,"end_diff":-33.5509,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/statsforvaltningens-expansion_GT10130"}],"GT10237":[{"start":0,"end":352,"start_adjusted":13.2,"end_adjusted":272.5,"speaker":"Thomas Östros","party":"S","speaker_audio_meta":"Thomas Östros (S)","anftext_short":"Herr talman! Hans Backman har frågat mig vilka åtgärder jag avser att vidta för att skapa förutsättningar för fler nya...","start_diff":13.2891,"end_diff":-79.8653,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/foretagspolitik_GT10237"},{"start":262,"end":610,"start_adjusted":277,"end_adjusted":531.7,"speaker":"Hans Backman","party":"FP","speaker_audio_meta":"Hans Backman (Fp)","anftext_short":"Herr talman! Tack för svaret! Jag kan däremot inte dela näringsministerns positiva syn på företagarnas villkor i Sverige. När jag...","start_diff":15.1128,"end_diff":-78.7159,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/foretagspolitik_GT10237"},{"start":522,"end":875,"start_adjusted":539.4,"end_adjusted":796.7,"speaker":"Per Bill","party":"M","speaker_audio_meta":"Per Bill (M)","anftext_short":"Herr talman! Det här är två debatter som tangerar varandra och går i varandra lite grann. Jag tänkte se om...","start_diff":17.4853,"end_diff":-78.7447,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/foretagspolitik_GT10237"},{"start":786,"end":1136,"start_adjusted":800.9,"end_adjusted":1050.6,"speaker":"Thomas Östros","party":"S","speaker_audio_meta":"Thomas Östros (S)","anftext_short":"Herr talman! Jag uppskattar verkligen Per Bills ansats till en seriös debatt. Det är alltid bäst så. Vi har en...","start_diff":15.0478,"end_diff":-85.7928,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/foretagspolitik_GT10237"},{"start":1045,"end":1384,"start_adjusted":1059.6,"end_adjusted":1304.7,"speaker":"Hans Backman","party":"FP","speaker_audio_meta":"Hans Backman (Fp)","anftext_short":"Herr talman! Detta är som sagt en mycket viktig debatt eftersom det är många människor som berörs av dessa frågor...","start_diff":14.7078,"end_diff":-79.6553,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/foretagspolitik_GT10237"},{"start":1296,"end":1522,"start_adjusted":1311.9,"end_adjusted":1443.7,"speaker":"Per Bill","party":"M","speaker_audio_meta":"Per Bill (M)","anftext_short":"Herr talman! Tack, näringsministern, för en hederlig och intressant debatt. Jag tror att näringsministern har alldeles rätt i att en...","start_diff":16.0228,"end_diff":-78.6559,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/foretagspolitik_GT10237"},{"start":1434,"end":1783,"start_adjusted":1449.6,"end_adjusted":1704,"speaker":"Thomas Östros","party":"S","speaker_audio_meta":"Thomas Östros (S)","anftext_short":"Herr talman! Det som är sorgligt med er i högeralliansen är att ni inte ser vad det är som gör...","start_diff":15.6722,"end_diff":-79.3759,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/foretagspolitik_GT10237"},{"start":1695,"end":1935,"start_adjusted":1710.6,"end_adjusted":1856.5,"speaker":"Hans Backman","party":"FP","speaker_audio_meta":"Hans Backman (Fp)","anftext_short":"Herr talman! Jag får tacka näringsministern för debatten trots att han nu har förfallit till att ge sig på högeralliansen...","start_diff":15.7116,"end_diff":-78.8934,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/foretagspolitik_GT10237"},{"start":1845,"end":2078,"start_adjusted":1861,"end_adjusted":1984.5,"speaker":"Thomas Östros","party":"S","speaker_audio_meta":"Thomas Östros (S)","anftext_short":"Herr talman! Hans Backman snårar in sig i statistiken på ett sådant sätt att jag tror att han aldrig kan...","start_diff":16.0509,"end_diff":-93.8797,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/foretagspolitik_GT10237"}],"GV01SfU6":[{"start":0,"end":741,"start_adjusted":16.5,"end_adjusted":648.2,"speaker":"Göte Wahlström","party":"S","speaker_audio_meta":"Göte Wahlström (S)","anftext_short":"Fru talman! Dagens debatt handlar om socialförsäkringsutskottets betänkande 2007/08:SfU6 Migration och asylpolitik . Betänkandet behandlar motioner från allmänna motionstiden. Härigenom...","start_diff":16.6472,"end_diff":-93.1772,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":654,"end":1757,"start_adjusted":668,"end_adjusted":1664,"speaker":"Kalle Larsson","party":"V","speaker_audio_meta":"Kalle Larsson (V)","anftext_short":"Fru talman! Vi lever alltjämt i en orättvis och orolig världsordning. Vi lever i en värld där hundratusentals miljoner människor...","start_diff":14.1234,"end_diff":-93.4034,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":1669,"end":2509,"start_adjusted":1665.3,"end_adjusted":2427.3,"speaker":"Bodil Ceballos","party":"MP","speaker_audio_meta":"Bodil Ceballos (Mp)","anftext_short":"Fru talman! Det är aldrig lätt att komma efter Kalle Larsson som alltid säger ungefär detsamma som jag själv har...","start_diff":-3.5978,"end_diff":-82.0797,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":2421,"end":3099,"start_adjusted":2439.8,"end_adjusted":3009.1,"speaker":"Mikael Cederbratt","party":"M","speaker_audio_meta":"Mikael Cederbratt (M)","anftext_short":"Fru talman! Jag vill börja med att yrka bifall till förslaget i betänkandet och avslag på samtliga motioner. Migration är...","start_diff":18.8972,"end_diff":-90.2972,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":3012,"end":3315,"start_adjusted":3025,"end_adjusted":3143.7,"speaker":"Kalle Larsson","party":"V","speaker_audio_meta":"Kalle Larsson (V)","anftext_short":"Fru talman! Mikael Cederbratt försökte göra sig lustig och sade att vi agerade som om det var de som satt...","start_diff":13.0716,"end_diff":-171.7191,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":3136,"end":3315,"start_adjusted":3147.5,"end_adjusted":3234.1,"speaker":"Mikael Cederbratt","party":"M","speaker_audio_meta":"Mikael Cederbratt (M)","anftext_short":"Fru talman! Jag har lite svårt att förstå logiken i Kalle Larssons resonemang när det gäller EU. Om man inte...","start_diff":11.6009,"end_diff":-81.2522,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":3227,"end":3437,"start_adjusted":3238.9,"end_adjusted":3358.3,"speaker":"Kalle Larsson","party":"V","speaker_audio_meta":"Kalle Larsson (V)","anftext_short":"Fru talman! Vänsterpartiet var det parti som först av alla föreslog att vi skulle avskaffa Utlänningsnämnden och införa en annan...","start_diff":12.0297,"end_diff":-79.1366,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":3349,"end":3556,"start_adjusted":3361.1,"end_adjusted":3470.3,"speaker":"Mikael Cederbratt","party":"M","speaker_audio_meta":"Mikael Cederbratt (M)","anftext_short":"Fru talman! Det var bra med klarläggandet från Kalle Larsson om att den ordning vi nu har ändå är bra....","start_diff":12.2047,"end_diff":-86.1034,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":3468,"end":3729,"start_adjusted":3471.6,"end_adjusted":3609.9,"speaker":"Bodil Ceballos","party":"MP","speaker_audio_meta":"Bodil Ceballos (Mp)","anftext_short":"Fru talman! Jag vill också betona att den nya ordningen är bättre än den gamla. Men när man kommer till...","start_diff":3.6853,"end_diff":-119.4966,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":3598,"end":3803,"start_adjusted":3609.5,"end_adjusted":3648.8,"speaker":"Mikael Cederbratt","party":"M","speaker_audio_meta":"Mikael Cederbratt (M)","anftext_short":"Fru talman! Det var bra att Bodil sade att det kan vara en poäng att ändå vara med i EU...","start_diff":11.5878,"end_diff":-154.6334,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":3643,"end":3803,"start_adjusted":3651,"end_adjusted":3727.4,"speaker":"Bodil Ceballos","party":"MP","speaker_audio_meta":"Bodil Ceballos (Mp)","anftext_short":"Fru talman! Jag tror att en gemensam signal från oss alla partier till domstolarna redan i dag om att tolkningen...","start_diff":8.1003,"end_diff":-75.9622,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":3719,"end":3926,"start_adjusted":3727.8,"end_adjusted":3836.6,"speaker":"Mikael Cederbratt","party":"M","speaker_audio_meta":"Mikael Cederbratt (M)","anftext_short":"Fru talman! Vi har den ordningen i Sverige när det gäller domstolarna att de är suveräna. Det innebär att domstolar...","start_diff":8.8816,"end_diff":-89.7641,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":3839,"end":4105,"start_adjusted":3849,"end_adjusted":3976.4,"speaker":"Göte Wahlström","party":"S","speaker_audio_meta":"Göte Wahlström (S)","anftext_short":"Fru talman! Som företrädare för det största regeringsbärande partiet har Mikael Cederbratt en mycket försiktig hållning. Det är helt naturligt....","start_diff":10.1284,"end_diff":-129.0391,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":3966,"end":4105,"start_adjusted":3978.5,"end_adjusted":4024.3,"speaker":"Mikael Cederbratt","party":"M","speaker_audio_meta":"Mikael Cederbratt (M)","anftext_short":"Fru talman! Det händer mycket. Jag har ju inte hela historiebeskrivningen så som Göte har. Ni glömde i alla fall...","start_diff":12.5934,"end_diff":-81.1141,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":4019,"end":4311,"start_adjusted":4028.3,"end_adjusted":4150.8,"speaker":"Göte Wahlström","party":"S","speaker_audio_meta":"Göte Wahlström (S)","anftext_short":"Fru talman! För att börja med historiken är det ett antal månader sedan vi lyfte upp frågan i socialförsäkringsutskottet. Signalerna...","start_diff":9.4422,"end_diff":-160.5684,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":4142,"end":4311,"start_adjusted":4153.3,"end_adjusted":4221,"speaker":"Mikael Cederbratt","party":"M","speaker_audio_meta":"Mikael Cederbratt (M)","anftext_short":"Fru talman! Jag tolkar ändå Göte Wahlströms svar på det sättet att om man skulle följa den här motionen om...","start_diff":11.4184,"end_diff":-90.4359,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":4223,"end":4949,"start_adjusted":4241.3,"end_adjusted":4861.3,"speaker":"Fredrick Federley","party":"C","speaker_audio_meta":"Fredrick Federley (C)","anftext_short":"Fru talman! Vi befinner oss i minst sagt intressanta tider när det gäller migrationspolitiken. Det handlar om hur det ser...","start_diff":18.4384,"end_diff":-88.0803,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":4863,"end":5157,"start_adjusted":4875.9,"end_adjusted":5005.6,"speaker":"Kalle Larsson","party":"V","speaker_audio_meta":"Kalle Larsson (V)","anftext_short":"Fru talman! Låt mig allra först tillåta mig själv att påminna Fredrick Federley om den omfattande och grundläggande kritik som...","start_diff":12.9722,"end_diff":-151.7822,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":4999,"end":5157,"start_adjusted":5008.3,"end_adjusted":5079,"speaker":"Fredrick Federley","party":"C","speaker_audio_meta":"Fredrick Federley (C)","anftext_short":"Fru talman! Min direkta fråga till Kalle Larsson är: I vilka andra ärenden som ska gå till domstol tycker du...","start_diff":9.3734,"end_diff":-78.3591,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":5069,"end":5378,"start_adjusted":5079.7,"end_adjusted":5211.7,"speaker":"Kalle Larsson","party":"V","speaker_audio_meta":"Kalle Larsson (V)","anftext_short":"Fru talman! Att Fredrick Federley är tvungen att ta till Socialdemokraterna som argument mot mig visar väl hur svag hans...","start_diff":10.8391,"end_diff":-166.7216,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":5201,"end":5378,"start_adjusted":5213,"end_adjusted":5290.3,"speaker":"Fredrick Federley","party":"C","speaker_audio_meta":"Fredrick Federley (C)","anftext_short":"Fru talman! Det enda som Kalle Larssons argumentation visar är hur svagt hans stöd är i folkopinionen men också här...","start_diff":12.0503,"end_diff":-88.0841,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":5291,"end":5492,"start_adjusted":5301.3,"end_adjusted":5412.8,"speaker":"Göte Wahlström","party":"S","speaker_audio_meta":"Göte Wahlström (S)","anftext_short":"Fru talman! Fredrick Federley har nu vid ett otal tillfällen sagt hur viktig den individuella prövningen är. För några år...","start_diff":10.3741,"end_diff":-79.5716,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":5407,"end":5604,"start_adjusted":5416.2,"end_adjusted":5521.3,"speaker":"Fredrick Federley","party":"C","speaker_audio_meta":"Fredrick Federley (C)","anftext_short":"Fru talman! Jag tackar Göte för frågan. Under den tid som debatten fördes om att införa en ny instans- och...","start_diff":9.2928,"end_diff":-83.1328,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":5516,"end":5733,"start_adjusted":5526.8,"end_adjusted":5653.5,"speaker":"Göte Wahlström","party":"S","speaker_audio_meta":"Göte Wahlström (S)","anftext_short":"Fru talman! Då var det rätt, säger Fredrick Federley. Vi kan inte bedriva kvartalspolitik inom migrationspolitiken. Det är viktigt att...","start_diff":10.9422,"end_diff":-79.9003,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":5777,"end":5993,"start_adjusted":5778,"end_adjusted":5917.1,"speaker":"Bodil Ceballos","party":"MP","speaker_audio_meta":"Bodil Ceballos (Mp)","anftext_short":"Fru talman! Jag vill börja med att säga till Fredrick Federley att jag tycker att den spanska modellen är bra....","start_diff":1.1097,"end_diff":-76.2959,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":5907,"end":6095,"start_adjusted":5917.1,"end_adjusted":6015.7,"speaker":"Fredrick Federley","party":"C","speaker_audio_meta":"Fredrick Federley (C)","anftext_short":"Fru talman! Tack, Bodil! Jag tycker faktiskt att du blir svaret skyldig här. Varför ska vi ha en individuell prövning,...","start_diff":10.1766,"end_diff":-79.6953,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":6009,"end":6221,"start_adjusted":6016.4,"end_adjusted":6145.1,"speaker":"Bodil Ceballos","party":"MP","speaker_audio_meta":"Bodil Ceballos (Mp)","anftext_short":"Fru talman! Jag kan börja med den sista frågan. Vi instämmer inte i Socialdemokraternas synpunkter i den delen. Jag vet...","start_diff":7.5197,"end_diff":-76.3316,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":6133,"end":6355,"start_adjusted":6145.2,"end_adjusted":6260.9,"speaker":"Fredrick Federley","party":"C","speaker_audio_meta":"Fredrick Federley (C)","anftext_short":"Fru talman! I sitt huvudanförande sade Bodil Ceballos att man behöver hitta en lösning på detta, och det bör man...","start_diff":12.3266,"end_diff":-94.5184,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":6268,"end":7106,"start_adjusted":6278.5,"end_adjusted":7018.5,"speaker":"Ulf Nilsson","party":"FP","speaker_audio_meta":"Ulf Nilsson (Fp)","anftext_short":"Fru talman! Vi läser ständigt nyheter och nås av rapporter och så vidare om människor som vistats länge i Sverige...","start_diff":10.5547,"end_diff":-87.9491,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":7020,"end":7235,"start_adjusted":7032,"end_adjusted":7156.8,"speaker":"Kalle Larsson","party":"V","speaker_audio_meta":"Kalle Larsson (V)","anftext_short":"Fru talman! Låt mig inledningsvis ta död på en fördom. Det är fördomen om att det skulle finnas en allmän...","start_diff":12.1078,"end_diff":-78.6247,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":7147,"end":7363,"start_adjusted":7159.1,"end_adjusted":7284.7,"speaker":"Ulf Nilsson","party":"FP","speaker_audio_meta":"Ulf Nilsson (Fp)","anftext_short":"Fru talman! Den viktigaste delen av Kalle Larssons fråga är egentligen inte vad som hände tidigare utan hur vi ser...","start_diff":12.1934,"end_diff":-78.6616,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":7278,"end":7555,"start_adjusted":7287.7,"end_adjusted":7408.8,"speaker":"Kalle Larsson","party":"V","speaker_audio_meta":"Kalle Larsson (V)","anftext_short":"Fru talman! Nu svajar argumentationen betänkligt. Avslutningsvis säger Ulf Nilsson att hur man än tolkar begreppet väpnad konflikt ger Sverige...","start_diff":9.7809,"end_diff":-146.5797,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":7402,"end":7555,"start_adjusted":7410.9,"end_adjusted":7466.3,"speaker":"Ulf Nilsson","party":"FP","speaker_audio_meta":"Ulf Nilsson (Fp)","anftext_short":"Fru talman! Det är ett ganska komplicerat arbete att se över en lagstiftning. Det enkla svaret är det som flera...","start_diff":8.9853,"end_diff":-89.1372,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":7468,"end":7687,"start_adjusted":7479.7,"end_adjusted":7610.4,"speaker":"Göte Wahlström","party":"S","speaker_audio_meta":"Göte Wahlström (S)","anftext_short":"Fru talman! Jag var lite anklagande mot Centerpartiet tidigare och sade att man var historielös. Men jag måste säga att...","start_diff":11.7678,"end_diff":-77.0078,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":7598,"end":7812,"start_adjusted":7609.9,"end_adjusted":7734.5,"speaker":"Ulf Nilsson","party":"FP","speaker_audio_meta":"Ulf Nilsson (Fp)","anftext_short":"Fru talman! Folkpartiet har varit mycket aktigt i migrations- och integrationsdebatten. Inte minst hade vi det som ett av våra...","start_diff":11.9922,"end_diff":-77.8753,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":7724,"end":8000,"start_adjusted":7735.6,"end_adjusted":7864.3,"speaker":"Göte Wahlström","party":"S","speaker_audio_meta":"Göte Wahlström (S)","anftext_short":"Fru talman! Historielösheten är större än vad jag hade förväntat mig från Ulf Nilssons sida. Om man läser protokollen och...","start_diff":11.7278,"end_diff":-136.1066,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":7852,"end":8000,"start_adjusted":7864,"end_adjusted":7915.2,"speaker":"Ulf Nilsson","party":"FP","speaker_audio_meta":"Ulf Nilsson (Fp)","anftext_short":"Fru talman! Jag var väldigt aktiv redan i valrörelsen 2002, och jag minns hur Socialdemokraterna alltid har svajat på den...","start_diff":12.1297,"end_diff":-85.1609,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":7912,"end":8128,"start_adjusted":7925.7,"end_adjusted":8050,"speaker":"Bodil Ceballos","party":"MP","speaker_audio_meta":"Bodil Ceballos (Mp)","anftext_short":"Fru talman! Jag vill börja med att säga att det är väldigt mycket tjat om migrationspolitiska talesmän här, men faktum...","start_diff":13.8247,"end_diff":-78.4141,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":8044,"end":8239,"start_adjusted":8052.7,"end_adjusted":8160.8,"speaker":"Ulf Nilsson","party":"FP","speaker_audio_meta":"Ulf Nilsson (Fp)","anftext_short":"Fru talman! Vi hamnar i retoriska diskussioner här, men det är helt rätt som Bodil Ceballos säger, att vi ska...","start_diff":8.7922,"end_diff":-78.5622,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":8154,"end":8341,"start_adjusted":8162.8,"end_adjusted":8262.6,"speaker":"Bodil Ceballos","party":"MP","speaker_audio_meta":"Bodil Ceballos (Mp)","anftext_short":"Fru talman! Jag talar inte om enskilda fall. Jag tror att det är 1 700 irakier som väntar på utvisning,...","start_diff":8.9184,"end_diff":-78.7722,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":8256,"end":8462,"start_adjusted":8263.4,"end_adjusted":8368.5,"speaker":"Ulf Nilsson","party":"FP","speaker_audio_meta":"Ulf Nilsson (Fp)","anftext_short":"Fru talman! När det gäller arbetskraftsinvandring har det för mig som folkpartist varit en väldigt viktig fråga under de senaste...","start_diff":7.5103,"end_diff":-93.9491,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":8374,"end":8883,"start_adjusted":8400.4,"end_adjusted":8796.3,"speaker":"Lars Gustafsson","party":"KD","speaker_audio_meta":"Lars Gustafsson (Kd)","anftext_short":"Fru talman! Som framgått av debatten är det få frågor som spänner över så många skilda områden som migration. Mitt...","start_diff":26.5016,"end_diff":-87.0666,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":8796,"end":9001,"start_adjusted":8809.6,"end_adjusted":8921.9,"speaker":"Bodil Ceballos","party":"MP","speaker_audio_meta":"Bodil Ceballos (Mp)","anftext_short":"Fru talman! Lars Gustafsson och jag brukar vara på samma sida när det gäller att värna flyktingars och människors rättigheter....","start_diff":13.7034,"end_diff":-79.5334,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":8914,"end":9125,"start_adjusted":8924.6,"end_adjusted":9047.1,"speaker":"Lars Gustafsson","party":"KD","speaker_audio_meta":"Lars Gustafsson (Kd)","anftext_short":"Fru talman! När sådana fall dyker upp har jag ofta fått frågan från medierna vad jag anser. Jag säger att...","start_diff":10.7066,"end_diff":-78.3041,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":9037,"end":9302,"start_adjusted":9047.4,"end_adjusted":9160.4,"speaker":"Bodil Ceballos","party":"MP","speaker_audio_meta":"Bodil Ceballos (Mp)","anftext_short":"Fru talman! Då måste jag ställa en följdfråga. Miljöpartiet föreslår nämligen att Sverige ska ha ett uppföljningsansvar för dem som...","start_diff":10.5059,"end_diff":-142.0391,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":9152,"end":9302,"start_adjusted":9163.1,"end_adjusted":9212.7,"speaker":"Lars Gustafsson","party":"KD","speaker_audio_meta":"Lars Gustafsson (Kd)","anftext_short":"Fru talman! När det gäller det sista som togs upp vill jag säga att det finns anledning att se över...","start_diff":11.2347,"end_diff":-89.6759,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":9214,"end":9486,"start_adjusted":9228,"end_adjusted":9343.7,"speaker":"Göte Wahlström","party":"S","speaker_audio_meta":"Göte Wahlström (S)","anftext_short":"Fru talman! Lars Gustafsson inledde sitt anförande med ett viktigt begrepp, nämligen att alla har lika värde. Det står vi...","start_diff":14.1359,"end_diff":-142.7259,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":9333,"end":9586,"start_adjusted":9346.4,"end_adjusted":9407,"speaker":"Lars Gustafsson","party":"KD","speaker_audio_meta":"Lars Gustafsson (Kd)","anftext_short":"Fru talman! Den är mycket enkel. Jag tror nämligen inte att man kan lagstifta retroaktivt. Var skulle vi i så...","start_diff":13.5478,"end_diff":-179.3941,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":9401,"end":9661,"start_adjusted":9409.7,"end_adjusted":9507.5,"speaker":"Göte Wahlström","party":"S","speaker_audio_meta":"Göte Wahlström (S)","anftext_short":"Fru talman! Min frågeställning är förknippad med att en partikollega till Lars Gustafsson, också utskottsersättare, säger att för Kristdemokraternas del...","start_diff":8.8291,"end_diff":-153.9203,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":9501,"end":9661,"start_adjusted":9510.1,"end_adjusted":9573.1,"speaker":"Lars Gustafsson","party":"KD","speaker_audio_meta":"Lars Gustafsson (Kd)","anftext_short":"Fru talman! Jag tror att Göte Wahlström fullständigt har missförstått alltihop. Möjligheten gällde ju att man får söka ett jobb,...","start_diff":9.1509,"end_diff":-88.2597,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":9575,"end":10268,"start_adjusted":9594.5,"end_adjusted":10179.7,"speaker":"Tobias Billström","party":"M","speaker_audio_meta":"Tobias Billström (M)","anftext_short":"Fru talman! Efter att ha lyssnat på det som har framförts under dagens debatt kan jag bara konstatera att det...","start_diff":19.6272,"end_diff":-88.7384,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":10181,"end":10394,"start_adjusted":10194.8,"end_adjusted":10315.4,"speaker":"Kalle Larsson","party":"V","speaker_audio_meta":"Kalle Larsson (V)","anftext_short":"Fru talman! Först har jag ett par kommentarer, och sedan ett par frågor. När migrationsministern så tydligt säger vad som...","start_diff":13.8709,"end_diff":-78.9959,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":10305,"end":10515,"start_adjusted":10317.9,"end_adjusted":10436.6,"speaker":"Tobias Billström","party":"M","speaker_audio_meta":"Tobias Billström (M)","anftext_short":"Fru talman! Det är bra att Kalle Larsson så tydligt uppfattar budskapet från regeringen när det gäller frågan om de...","start_diff":12.9741,"end_diff":-78.8334,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":10429,"end":10643,"start_adjusted":10440.9,"end_adjusted":10564.5,"speaker":"Kalle Larsson","party":"V","speaker_audio_meta":"Kalle Larsson (V)","anftext_short":"Fru talman! Då kan väl Tobias Billström i sin andra replik också förklara, om han tycker att det är okej...","start_diff":11.9591,"end_diff":-78.9041,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":10554,"end":10771,"start_adjusted":10567.5,"end_adjusted":10687.4,"speaker":"Tobias Billström","party":"M","speaker_audio_meta":"Tobias Billström (M)","anftext_short":"Fru talman! Man kan kanske börja med att sätta sig in i hur vi har organiserat oss här i Sverige,...","start_diff":13.6397,"end_diff":-84.0034,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":10685,"end":10898,"start_adjusted":10696.5,"end_adjusted":10819.3,"speaker":"Bodil Ceballos","party":"MP","speaker_audio_meta":"Bodil Ceballos (Mp)","anftext_short":"Fru talman! Dagens stora fråga har varit den väpnade konflikten och möjligheten för oss att göra skillnad. I somras var...","start_diff":11.6322,"end_diff":-79.0578,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":10813,"end":11112,"start_adjusted":10822.3,"end_adjusted":10943,"speaker":"Tobias Billström","party":"M","speaker_audio_meta":"Tobias Billström (M)","anftext_short":"Fru talman! En sak är säker: Tilltron till system uppnår man i varje fall inte genom att ständigt och jämt...","start_diff":9.3509,"end_diff":-169.4147,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":10934,"end":11112,"start_adjusted":10945.7,"end_adjusted":11031.6,"speaker":"Bodil Ceballos","party":"MP","speaker_audio_meta":"Bodil Ceballos (Mp)","anftext_short":"Fru talman! Jag vill ställa en annan fråga, som handlar om den aktiva återvändandepolitiken. Det står i Migrationsverkets regleringsbrev att...","start_diff":11.8422,"end_diff":-80.7534,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":11027,"end":11246,"start_adjusted":11035.5,"end_adjusted":11158.6,"speaker":"Tobias Billström","party":"M","speaker_audio_meta":"Tobias Billström (M)","anftext_short":"Fru talman! Regeringen i Irak har varit en varm tillskyndare av återtagandeavtalet. Det märkte jag inte minst själv när jag...","start_diff":8.6172,"end_diff":-87.7691,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":11161,"end":11362,"start_adjusted":11169.7,"end_adjusted":11282.6,"speaker":"Göte Wahlström","party":"S","speaker_audio_meta":"Göte Wahlström (S)","anftext_short":"Fru talman! Först får jag säga att jag är väldigt glad att statsrådet är här i kammaren, lyssnar till debatten...","start_diff":8.7566,"end_diff":-79.7547,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":11273,"end":11475,"start_adjusted":11285.4,"end_adjusted":11396.6,"speaker":"Tobias Billström","party":"M","speaker_audio_meta":"Tobias Billström (M)","anftext_short":"Fru talman! Det är alltid viktigt och angeläget att föra en diskussion även om naturligtvis regeringsunderlaget är fullständigt klart i...","start_diff":12.5359,"end_diff":-78.7978,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":11387,"end":11607,"start_adjusted":11397.8,"end_adjusted":11526.1,"speaker":"Göte Wahlström","party":"S","speaker_audio_meta":"Göte Wahlström (S)","anftext_short":"Fru talman! Jag inledde med att säga att det finns en fara i att retoriken blockerar. Det är kanske så....","start_diff":10.9234,"end_diff":-81.2822,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"},{"start":11521,"end":11738,"start_adjusted":11529.3,"end_adjusted":11627.5,"speaker":"Tobias Billström","party":"M","speaker_audio_meta":"Tobias Billström (M)","anftext_short":"Fru talman! Ljudanläggningen var det. Jag tyckte att jag var mycket tydlig i talarstolen när jag sade att regeringen inte...","start_diff":8.4303,"end_diff":-110.9478,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/migration-och-asylpolitik_GV01SfU6"}],"GW10399":[{"start":0,"end":298,"start_adjusted":9.5,"end_adjusted":208.2,"speaker":"Mats Odell","party":"KD","speaker_audio_meta":"Mats Odell (Kd)","anftext_short":"Fru talman! Bosse Ringholm har frågat mig vilka nya åtgärder jag har för avsikt att vidta för att ta itu...","start_diff":9.6441,"end_diff":-90.1928,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/brist-pa-bostader-i-stockholm_GW10399"},{"start":211,"end":553,"start_adjusted":220.6,"end_adjusted":463.7,"speaker":"Bosse Ringholm","party":"S","speaker_audio_meta":"Bosse Ringholm (S)","anftext_short":"Fru talman! Jag tackar statsrådet för svaret. Bostadsbristen i Stockholm är förvisso inte ny, men den har vuxit kraftigt det...","start_diff":9.6828,"end_diff":-89.7391,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/brist-pa-bostader-i-stockholm_GW10399"},{"start":467,"end":778,"start_adjusted":474.8,"end_adjusted":692.7,"speaker":"Andreas Norlén","party":"M","speaker_audio_meta":"Andreas Norlén (M)","anftext_short":"Fru talman! Det här är ytterligare en bostadspolitisk debatt av många där Socialdemokraterna vill ge sken av att det knappt...","start_diff":7.9384,"end_diff":-85.7284,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/brist-pa-bostader-i-stockholm_GW10399"},{"start":690,"end":941,"start_adjusted":700.2,"end_adjusted":854.5,"speaker":"Mats Odell","party":"KD","speaker_audio_meta":"Mats Odell (Kd)","anftext_short":"Fru talman! Bosse Ringholm hänvisar ömsom till majoriteten i kammaren och ömsom till majoriteten i Stockholms stadshus för att belägga...","start_diff":10.2534,"end_diff":-86.9141,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/brist-pa-bostader-i-stockholm_GW10399"},{"start":853,"end":1132,"start_adjusted":865.1,"end_adjusted":968.1,"speaker":"Bosse Ringholm","party":"S","speaker_audio_meta":"Bosse Ringholm (S)","anftext_short":"Fru talman! När jag hör Mats Odell tänker jag på det gamla ordspråket Bättre fly än illa fäkta, men i...","start_diff":12.1728,"end_diff":-164.3116,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/brist-pa-bostader-i-stockholm_GW10399"},{"start":969,"end":1132,"start_adjusted":979,"end_adjusted":1046.5,"speaker":"Andreas Norlén","party":"M","speaker_audio_meta":"Andreas Norlén (M)","anftext_short":"Fru talman! Jag konstaterar att inte heller Bosse Ringholm gav någon större vägledning eller hade en speciellt bra kväll när...","start_diff":10.1297,"end_diff":-85.9272,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/brist-pa-bostader-i-stockholm_GW10399"},{"start":1044,"end":1341,"start_adjusted":1054,"end_adjusted":1208.1,"speaker":"Mats Odell","party":"KD","speaker_audio_meta":"Mats Odell (Kd)","anftext_short":"Fru talman! Åtgärden är egentligen kort och gott att riva upp den bostadspolitik som Socialdemokraterna har drivit som har lett...","start_diff":10.0884,"end_diff":-133.3153,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/brist-pa-bostader-i-stockholm_GW10399"},{"start":1253,"end":1420,"start_adjusted":1263.3,"end_adjusted":1321.6,"speaker":"Mats Odell","party":"KD","speaker_audio_meta":"Mats Odell (Kd)","anftext_short":"Fru talman! Bosse Ringholm lär inte godkänna svaren, vilka jag än kommer med. När jag kommer med fakta om bakgrunden...","start_diff":10.4059,"end_diff":-98.7972,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/brist-pa-bostader-i-stockholm_GW10399"}],"GW10502":[{"start":0,"end":215,"start_adjusted":15.6,"end_adjusted":131.7,"speaker":"Maud Olofsson","party":"C","speaker_audio_meta":"Maud Olofsson (C)","anftext_short":"Fru talman! Man slutar i riksdagen och börjar i riksdagen. Det var en intressant energi- och klimatdebatt i går. Leif...","start_diff":15.7191,"end_diff":-83.7041,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/elleverans-till-kommunala-anlaggningar_GW10502"},{"start":127,"end":425,"start_adjusted":142.8,"end_adjusted":338.1,"speaker":"Leif Pettersson","party":"S","speaker_audio_meta":"Leif Pettersson (S)","anftext_short":"Fru talman! Jag vill tacka ministern för svaret på min interpellation. Jag konstaterar att vi än en gång diskuterar frågan...","start_diff":15.9228,"end_diff":-87.2553,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/elleverans-till-kommunala-anlaggningar_GW10502"},{"start":337,"end":681,"start_adjusted":353.2,"end_adjusted":600.3,"speaker":"Karin Åström","party":"S","speaker_audio_meta":"Karin Åström (S)","anftext_short":"Fru talman! Jag tackar ministern för svaret och även riksdagsledamoten Leif Pettersson för att han har ställt en interpellation i...","start_diff":16.3372,"end_diff":-81.0516,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/elleverans-till-kommunala-anlaggningar_GW10502"},{"start":591,"end":871,"start_adjusted":602.7,"end_adjusted":789.3,"speaker":"Maud Olofsson","party":"C","speaker_audio_meta":"Maud Olofsson (C)","anftext_short":"Fru talman! Jag tror att man måste skilja på två saker. Den ena handlar om gatubelysningen, den andra handlar om...","start_diff":11.7666,"end_diff":-82.1359,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/elleverans-till-kommunala-anlaggningar_GW10502"},{"start":783,"end":1093,"start_adjusted":798.1,"end_adjusted":1006.9,"speaker":"Leif Pettersson","party":"S","speaker_audio_meta":"Leif Pettersson (S)","anftext_short":"Fru talman! Det är faktiskt så att Haparanda kommun har överklagat beslutet om gatubelysningen. Sedan en månad tillbaka har minister...","start_diff":15.2297,"end_diff":-86.4822,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/elleverans-till-kommunala-anlaggningar_GW10502"},{"start":1005,"end":1199,"start_adjusted":1020.2,"end_adjusted":1119.8,"speaker":"Karin Åström","party":"S","speaker_audio_meta":"Karin Åström (S)","anftext_short":"Fru talman! I den senaste interpellationsdebatten jag hade med ministern i den här frågan, riksmötet 2007/2008, sade ministern att man...","start_diff":15.2709,"end_diff":-79.5547,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/elleverans-till-kommunala-anlaggningar_GW10502"},{"start":1110,"end":1375,"start_adjusted":1119.3,"end_adjusted":1293.2,"speaker":"Maud Olofsson","party":"C","speaker_audio_meta":"Maud Olofsson (C)","anftext_short":"Fru talman! Ni har fått både skärpt lagstiftning och skärpt tillämpning. Det är det som har hänt. Energimarknadsinspektionen har varit...","start_diff":9.4453,"end_diff":-82.2316,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/elleverans-till-kommunala-anlaggningar_GW10502"},{"start":1287,"end":1503,"start_adjusted":1301.3,"end_adjusted":1424,"speaker":"Leif Pettersson","party":"S","speaker_audio_meta":"Leif Pettersson (S)","anftext_short":"Fru talman! Skärpt lagstiftning har vi fått, men konsumenterna far fortfarande illa. Konsumenterna får fortfarande betala alldeles för höga avgifter....","start_diff":14.4422,"end_diff":-79.3828,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/elleverans-till-kommunala-anlaggningar_GW10502"},{"start":1415,"end":1623,"start_adjusted":1426.9,"end_adjusted":1530.7,"speaker":"Maud Olofsson","party":"C","speaker_audio_meta":"Maud Olofsson (C)","anftext_short":"Fru talman! Historien om konflikten i Haparanda och Övertorneå har jag fått lyssna till, så jag tror att jag har...","start_diff":11.9584,"end_diff":-92.7328,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/elleverans-till-kommunala-anlaggningar_GW10502"}],"GW1065":[{"start":0,"end":177,"start_adjusted":19.3,"end_adjusted":96.7,"speaker":"Maud Olofsson","party":"C","speaker_audio_meta":"Maud Olofsson (C)","anftext_short":"Herr talman! Börje Vestlund har frågat mig vilka åtgärder jag är beredd att vidta för att utveckla företag som sysslar...","start_diff":19.4147,"end_diff":-80.6859,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/design-i-svenska-foretag_GW1065"},{"start":90,"end":403,"start_adjusted":108.7,"end_adjusted":322.4,"speaker":"Börje Vestlund","party":"S","speaker_audio_meta":"Börje Vestlund (S)","anftext_short":"Herr talman! Jag ber att få tacka för svaret. Skälet till att jag har ställt interpellationen är ett förslag i...","start_diff":18.8353,"end_diff":-80.9997,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/design-i-svenska-foretag_GW1065"},{"start":316,"end":567,"start_adjusted":324.1,"end_adjusted":485.8,"speaker":"Maud Olofsson","party":"C","speaker_audio_meta":"Maud Olofsson (C)","anftext_short":"Herr talman! Jag tror att det finns en skiljelinje mellan Socialdemokraterna och regeringen när det gäller synen på hur man...","start_diff":8.1603,"end_diff":-81.6328,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/design-i-svenska-foretag_GW1065"},{"start":479,"end":774,"start_adjusted":491.9,"end_adjusted":694.4,"speaker":"Börje Vestlund","party":"S","speaker_audio_meta":"Börje Vestlund (S)","anftext_short":"Herr talman! Jag delar statsrådets uppfattning om att det finns en god vilja hos företagen att utvecklas och att göra...","start_diff":12.9822,"end_diff":-80.0241,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/design-i-svenska-foretag_GW1065"},{"start":685,"end":1030,"start_adjusted":695.2,"end_adjusted":873,"speaker":"Maud Olofsson","party":"C","speaker_audio_meta":"Maud Olofsson (C)","anftext_short":"Herr talman! Börje Vestlund säger att det ska vara träffsäkra åtgärder. Jag står här dag ut och dag in och...","start_diff":10.2753,"end_diff":-157.4191,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/design-i-svenska-foretag_GW1065"},{"start":866,"end":1120,"start_adjusted":875.5,"end_adjusted":948.2,"speaker":"Börje Vestlund","party":"S","speaker_audio_meta":"Börje Vestlund (S)","anftext_short":"Herr talman! Vi ska bygga stödfunktioner. Maud Olofsson säger att hon tittar på detta och att hon kanske kommer fram...","start_diff":9.6016,"end_diff":-172.2072,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/design-i-svenska-foretag_GW1065"},{"start":942,"end":1120,"start_adjusted":951.4,"end_adjusted":1031.5,"speaker":"Maud Olofsson","party":"C","speaker_audio_meta":"Maud Olofsson (C)","anftext_short":"Herr talman! Jag förstår om socialdemokrater blir upprörda när vi kritiserar er för att ha hur mycket pengar som helst....","start_diff":9.4716,"end_diff":-88.9122,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/design-i-svenska-foretag_GW1065"}],"GY01JuU25":[{"start":0,"end":865,"start_adjusted":19.1,"end_adjusted":768.8,"speaker":"Arhe Hamednaca","party":"S","speaker_audio_meta":"Arhe Hamednaca (S)","anftext_short":"Fru talman! Jag vill börja med att yrka bifall till vår reservation nr 6 och de gemensamma med Miljöpartiet och...","start_diff":19.1784,"end_diff":-96.6222,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/forbattrad-utslussning-fran-sluten-ungdomsvard_GY01JuU25"},{"start":777,"end":1457,"start_adjusted":790.4,"end_adjusted":1363.8,"speaker":"Mehmet Kaplan","party":"MP","speaker_audio_meta":"Mehmet Kaplan (Mp)","anftext_short":"Fru talman! Jag vill tacka Hamednaca för en bra genomgång av våra gemensamma reservationer. Naturligtvis står vi i Miljöpartiet bakom...","start_diff":13.4503,"end_diff":-93.5759,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/forbattrad-utslussning-fran-sluten-ungdomsvard_GY01JuU25"},{"start":1369,"end":1687,"start_adjusted":1379.8,"end_adjusted":1594.8,"speaker":"Marianne Berg","party":"V","speaker_audio_meta":"Marianne Berg (V)","anftext_short":"Fru talman! I detta betänkande vill jag belysa en punkt: punkt 6, Förbättrade kontrollmöjligheter. Fru talman! Elektronisk övervakning eller så...","start_diff":10.9278,"end_diff":-92.5572,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/forbattrad-utslussning-fran-sluten-ungdomsvard_GY01JuU25"},{"start":1599,"end":2449,"start_adjusted":1608.1,"end_adjusted":2272.9,"speaker":"Johan Linander","party":"C","speaker_audio_meta":"Johan Linander (C)","anftext_short":"Fru talman! I dag debatterar vi viktiga lagändringar för att fler unga som har dömts till sluten ungdomsvård ska kunna...","start_diff":9.1622,"end_diff":-176.5366,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/forbattrad-utslussning-fran-sluten-ungdomsvard_GY01JuU25"},{"start":2275,"end":2449,"start_adjusted":2287.3,"end_adjusted":2363.5,"speaker":"Marianne Berg","party":"V","speaker_audio_meta":"Marianne Berg (V)","anftext_short":"Fru talman! Att det inte finns någon planering tycker jag naturligtvis är illa, men det var inte för att hålla...","start_diff":12.4316,"end_diff":-85.8672,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/forbattrad-utslussning-fran-sluten-ungdomsvard_GY01JuU25"},{"start":2359,"end":2639,"start_adjusted":2365.9,"end_adjusted":2466.2,"speaker":"Johan Linander","party":"C","speaker_audio_meta":"Johan Linander (C)","anftext_short":"Fru talman! Jag beklagar om Marianne Berg inte lyssnade eller missuppfattade vad jag sade. Vad jag sade i mitt anförande...","start_diff":6.9678,"end_diff":-173.1997,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/forbattrad-utslussning-fran-sluten-ungdomsvard_GY01JuU25"},{"start":2463,"end":2639,"start_adjusted":2469.9,"end_adjusted":2554.7,"speaker":"Marianne Berg","party":"V","speaker_audio_meta":"Marianne Berg (V)","anftext_short":"Fru talman! Att sitta kvar inom sluten ungdomsvård är inte ett alternativ som vi har sagt ska finnas. Vi säger:...","start_diff":6.9516,"end_diff":-84.6566,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/forbattrad-utslussning-fran-sluten-ungdomsvard_GY01JuU25"},{"start":2551,"end":2841,"start_adjusted":2557.5,"end_adjusted":2653.4,"speaker":"Johan Linander","party":"C","speaker_audio_meta":"Johan Linander (C)","anftext_short":"Fru talman! Det är uppenbart att Marianne Berg inte har läst propositionen, för det här är en möjlighet. Vi tvingar...","start_diff":6.5834,"end_diff":-187.9547,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/forbattrad-utslussning-fran-sluten-ungdomsvard_GY01JuU25"},{"start":2657,"end":2925,"start_adjusted":2669.2,"end_adjusted":2757,"speaker":"Mehmet Kaplan","party":"MP","speaker_audio_meta":"Mehmet Kaplan (Mp)","anftext_short":"Fru talman! Tack, Johan Linander, för en tydlig genomgång av både egna och andras förslag! Jag vill fråga Johan Linander...","start_diff":12.2622,"end_diff":-168.4266,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/forbattrad-utslussning-fran-sluten-ungdomsvard_GY01JuU25"},{"start":2753,"end":2925,"start_adjusted":2759,"end_adjusted":2840.9,"speaker":"Johan Linander","party":"C","speaker_audio_meta":"Johan Linander (C)","anftext_short":"Fru talman! Jag hälsar Mehmet Kaplan välkommen tillbaka till justitiedebatterna om han har saknat dem. Men jag förstår att han...","start_diff":6.0878,"end_diff":-84.5241,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/forbattrad-utslussning-fran-sluten-ungdomsvard_GY01JuU25"},{"start":2837,"end":3131,"start_adjusted":2842.5,"end_adjusted":2967.7,"speaker":"Mehmet Kaplan","party":"MP","speaker_audio_meta":"Mehmet Kaplan (Mp)","anftext_short":"Fru talman! Jag tackar Johan Linander. I min fråga lyfte jag fram två delar. Det är naturligtvis välkommet att den...","start_diff":5.5684,"end_diff":-163.7422,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/forbattrad-utslussning-fran-sluten-ungdomsvard_GY01JuU25"},{"start":2963,"end":3131,"start_adjusted":2969.4,"end_adjusted":3037.9,"speaker":"Johan Linander","party":"C","speaker_audio_meta":"Johan Linander (C)","anftext_short":"Fru talman! Det räcker att läsa texten i förslaget till ändring i lagen om verkställighet av sluten ungdomsvård, 17 a...","start_diff":6.4516,"end_diff":-93.4578,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/forbattrad-utslussning-fran-sluten-ungdomsvard_GY01JuU25"},{"start":3044,"end":3509,"start_adjusted":3050.8,"end_adjusted":3410.7,"speaker":"Helena Bouveng","party":"M","speaker_audio_meta":"Helena Bouveng (M)","anftext_short":"Fru talman! I dag är det en efterlängtad lagändring som vi ska fatta beslut om. Vi som arbetar med de...","start_diff":6.9072,"end_diff":-98.6722,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/forbattrad-utslussning-fran-sluten-ungdomsvard_GY01JuU25"},{"start":3422,"end":4076,"start_adjusted":3429.3,"end_adjusted":3983,"speaker":"Johan Pehrson","party":"L","speaker_audio_meta":"Johan Pehrson (Fp)","anftext_short":"Fru talman! Propositionen och betänkandet behandlar innehållet i den statliga ungdomsvården. Det ska bli starkare styrning och stöd och ökade...","start_diff":7.3797,"end_diff":-93.4409,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/forbattrad-utslussning-fran-sluten-ungdomsvard_GY01JuU25"},{"start":3988,"end":4436,"start_adjusted":3999,"end_adjusted":4339.9,"speaker":"Caroline Szyber","party":"KD","speaker_audio_meta":"Caroline Szyber (Kd)","anftext_short":"Fru talman! Ett återkommande kriminalpolitiskt debattämne är hur samhället ska reagera på ungdomars brottslighet. Kristdemokraterna menar att straffet ska vara...","start_diff":11.1303,"end_diff":-96.4672,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/forbattrad-utslussning-fran-sluten-ungdomsvard_GY01JuU25"},{"start":4348,"end":4600,"start_adjusted":4355.6,"end_adjusted":4505.1,"speaker":"Kent Ekeroth","party":"SD","speaker_audio_meta":"Kent Ekeroth (Sd)","anftext_short":"Fru talman! Nu hamnade jag sist här med en talartid på fyra minuter. Det beror på att jag inte blev...","start_diff":7.6653,"end_diff":-95.2778,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/forbattrad-utslussning-fran-sluten-ungdomsvard_GY01JuU25"},{"start":4512,"end":4814,"start_adjusted":4521.8,"end_adjusted":4646.8,"speaker":"Johan Pehrson","party":"L","speaker_audio_meta":"Johan Pehrson (Fp)","anftext_short":"Fru talman! Jag vill ge Kent Ekeroth rätt. Av reservationen framgår det att konfrontationer ska vara frivilliga, och det var...","start_diff":9.8841,"end_diff":-167.5616,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/forbattrad-utslussning-fran-sluten-ungdomsvard_GY01JuU25"},{"start":4642,"end":4858,"start_adjusted":4648.1,"end_adjusted":4729.9,"speaker":"Kent Ekeroth","party":"SD","speaker_audio_meta":"Kent Ekeroth (Sd)","anftext_short":"Fru talman! Precis som Johan Pehrson gjorde ett misstag när han kritiserade oss gjorde jag ett misstag när jag tyvärr...","start_diff":6.1766,"end_diff":-128.5028,"debateurl":"/sv/webb-tv/video/debatt-om-forslag/forbattrad-utslussning-fran-sluten-ungdomsvard_GY01JuU25"}],"GY10120":[{"start":0,"end":408,"start_adjusted":334.7,"end_adjusted":587.6,"speaker":"Suzanne Svensson","party":"S","speaker_audio_meta":"Andreas Carlgren (C)","anftext_short":"Fru talman! Jag vill först tacka minister Carlgren för svaret i dag. Jag vill också passa på att tacka för...","start_diff":334.8422,"end_diff":179.2078,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/odling-av-specialgrodor_GY10120"},{"start":318,"end":668,"start_adjusted":592.1,"end_adjusted":837.2,"speaker":"Andreas Carlgren","party":"C","speaker_audio_meta":"Suzanne Svensson (S)","anftext_short":"Fru talman! Suzanne Svenssons utgångspunkt är att det är viktigt med svensk odling. Där är vi sannerligen överens. Är det...","start_diff":274.1522,"end_diff":168.8059,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/odling-av-specialgrodor_GY10120"},{"start":580,"end":920,"start_adjusted":844.4,"end_adjusted":1055.9,"speaker":"Suzanne Svensson","party":"S","speaker_audio_meta":"Andreas Carlgren (C)","anftext_short":"Fru talman! Det känns bra med den inställningen. Det jag tycker är märkligt är att Danmark har valt att godkänna...","start_diff":264.4503,"end_diff":135.5397,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/odling-av-specialgrodor_GY10120"},{"start":832,"end":1146,"start_adjusted":1072.3,"end_adjusted":1206.5,"speaker":"Anders Åkesson","party":"C","speaker_audio_meta":"Suzanne Svensson (S)","anftext_short":"Fru talman! Låt mig inleda med att säga att jag delar den problemformulering som interpellanten Suzanne Svensson gör i sin...","start_diff":240.3809,"end_diff":60.1322,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/odling-av-specialgrodor_GY10120"},{"start":1058,"end":1286,"start_adjusted":1212.7,"end_adjusted":1443.7,"speaker":"Andreas Carlgren","party":"C","speaker_audio_meta":"Anders Åkesson (C)","anftext_short":"Fru talman! Jag vill konstatera att vi ju är väldigt överens i diskussionen. Det finns en mycket tydlig gemensam strävan...","start_diff":154.8484,"end_diff":157.3103,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/odling-av-specialgrodor_GY10120"},{"start":1198,"end":1614,"start_adjusted":1454.3,"end_adjusted":1528.1,"speaker":"Suzanne Svensson","party":"S","speaker_audio_meta":"Andreas Carlgren (C)","anftext_short":"Fru talman! Det är min första period i riksdagen, och man har alltid tyckt mycket om de olika verk som...","start_diff":256.3634,"end_diff":-86.2641,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/odling-av-specialgrodor_GY10120"},{"start":1443,"end":1690,"start_adjusted":1540.2,"end_adjusted":1610.3,"speaker":"Anders Åkesson","party":"C","speaker_audio_meta":"Suzanne Svensson (S)","anftext_short":"Fru talman! Jag tycker att miljöministerns besked att man fortsätter att hålla tryck på EU i denna fråga är hoppfullt,...","start_diff":97.3078,"end_diff":-80.1334,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/odling-av-specialgrodor_GY10120"},{"start":1526,"end":1757,"start_adjusted":1614.6,"end_adjusted":1659.5,"speaker":"Andreas Carlgren","party":"C","speaker_audio_meta":"Anders Åkesson (C)","anftext_short":"Fru talman! Jag vill tacka både Suzanne Svensson och Anders Åkesson för debatten. Jag tycker att det är bra att...","start_diff":88.6928,"end_diff":-97.9259,"debateurl":"/sv/webb-tv/video/interpellationsdebatt/odling-av-specialgrodor_GY10120"}]};</script>
</div>
</div>
<div class="column-page">
<div class="videobox-container">
  <div class="videobox">
    <iframe id="videokb_deb" src="https://www.riksdagen.se/sv/webb-tv/video/interpellationsdebatt/regeringens-skarpning-av-migrationspolitiken_gr10298/embed/?start=755.4&amp;end=1001.9" allowfullscreen="" scrolling="yes" title="Partiledardebatt 18 januari 2023 från Riksdagen om Partiledardebatt" style="width: 510px; height: 288px; border: 0´; margin-bottom: 0em;"></iframe>
    
    <div class="metabox_debate">
      <form action="">
        <input type="button" id="prevspeech3" value="Previous speech">
        <input type="button" id="nextspeech3" value="Next speech">
        <input type="button" id="prevdebate1" value="Previous debate" style="margin-left: 41px;">
        <input type="button" id="nextdebate1" value="Next debate">
      </form>
      <h3 class="anchored">Adjusted metadata (KBLab)</h3>
      <p class="beginning"><u>Beginning of official transcript:</u></p>
      <p id="speechtranscript3">Herr talman! Tyvärr tvingas jag notera att jag inte fick svar på de frågor jag ställde, men jag kan upprepa...</p>
      <p id="metadatastart3"><b>Start:</b> 00:12:35.4  <b>End:</b> 00:16:41.9</p>
      <p id="speaker3"><b>Speaker:</b> Erik Ullenhag (L)</p>
      <p><small><i>Source: Sveriges riksdag.</i></small></p>
    </div>
  </div>

  <div class="videobox">
    <iframe id="videoriks_deb" src="https://www.riksdagen.se/sv/webb-tv/video/interpellationsdebatt/regeringens-skarpning-av-migrationspolitiken_gr10298/embed/?start=0&amp;end=312" allowfullscreen="" scrolling="yes" title="Partiledardebatt 18 januari 2023 från Riksdagen om Partiledardebatt" style="width: 510px; height: 288px; border: 0; margin-bottom: 0em;"></iframe>
    <div class="metabox_debate">
      <form action="">
        <input type="button" id="prevspeech4" value="Previous speech">
        <input type="button" id="nextspeech4" value="Next speech">
        <input type="button" id="prevdebate2" value="Previous debate" style="margin-left: 41px;">
        <input type="button" id="nextdebate2" value="Next debate">
      </form>
      <h3 class="anchored">The Riksdag's metadata</h3>
      <p class="beginning"><u>Beginning of official transcript:</u></p>
      <p id="speechtranscript4">Herr talman! Tyvärr tvingas jag notera att jag inte fick svar på de frågor jag ställde, men jag kan upprepa...</p>
      <p id="metadatastart4"><b>Start:</b> 00:00:00.0  <b>End:</b> 00:05:12.0</p>
      <p id="speaker4"><b>Speaker:</b> Barbro Holmberg (S)</p>
      <p><small><i>Source: Sveriges riksdag.</i></small></p>
    </div>
  </div>
</div>
</div>
<p>A higher fraction of the earlier debates from 2003-2006 tend to start and end in the middle of speeches, looking like they may possibly have been automatically edited and spliced based on misaligned metadata. Several of the debates have issues with skips and cuts in the media, possibly resulting from video encoding errors upon conversion to web formats. Hopefully, the full debates are still available in an unedited format somewhere in the Riksdag’s archives.</p>
</section>
<section id="the-riksdags-speeches-in-numbers" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="the-riksdags-speeches-in-numbers">The Riksdag’s speeches in numbers</h2>
<p>The valid downloadable audio files from the Riksdag’s debates have a total duration 6361 hours. There is a metadata field called <code>debateseconds</code> indicating the duration of each debate in seconds. If we sum the claimed duration of debates they amount to 6398 hours.</p>
<table class="caption-top table">
<thead>
<tr class="header">
<th style="text-align: left;">Source</th>
<th style="text-align: right;">Total duration of debates (hours)</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">The Riksdag’s metadata</td>
<td style="text-align: right;">6398.4</td>
</tr>
<tr class="even">
<td style="text-align: left;">Audio files</td>
<td style="text-align: right;">6361.4</td>
</tr>
</tbody>
</table>
<p>How many of those 6360 hours are actually speeches though? According to our speech segmentation, the debate files consist of a total of 5858 hours of speeches. However, when looking at the metadata field <code>duration</code> (for individual speech durations), the total duration of speeches exceed the total duration of the audio files. We note the start and end indications of many speeches overlap with other speeches for metadata prior to 2012.</p>
<table class="caption-top table">
<thead>
<tr class="header">
<th style="text-align: left;">Source</th>
<th style="text-align: right;">Total speech duration of debates</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">The Riksdag’s metadata</td>
<td style="text-align: right;">6742.15</td>
</tr>
<tr class="even">
<td style="text-align: left;">Adjusted metadata (KBLab)</td>
<td style="text-align: right;">5858.36</td>
</tr>
</tbody>
</table>
<p>Our method for finding which debates had associated media files, was to first download the speeches in text form from <a href="https://data.riksdagen.se/data/anforanden/">“Riksdagen’s anföranden”</a>. We then queried the Riksdag’s media API, using the document ids of all those speeches. Out of more than 300000 available speeches from 1993/94 and forward:</p>
<ul>
<li><strong>133130</strong> speeches belonged to debates that had downloadable audio files in the Riksdag’s media API.</li>
<li>Of the above only <strong>122525</strong> speeches had valid audio files, or were found to be at all present in the audio files.</li>
<li>After applying additional quality filters <strong>117725</strong> speeches remained. These filters included removing:
<ul>
<li>duplicate transcripts attributed to different speakers.</li>
<li>two or more separate speeches being attributed with the same starting or ending time (indicating the model had failed making a valid prediction).</li>
<li>debates starting and ending in the middle of speeches.</li>
<li>sudden jumps/cuts/edits in the audio while a speech was in progress.</li>
<li>speeches shorter than 25 seconds in duration (the margins of error are narrower for shorter speeches).</li>
</ul></li>
</ul>
<section id="metadata-statistics-grouped-by-year" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="metadata-statistics-grouped-by-year">Metadata statistics grouped by year</h3>
<p>To get an overview of metadata quality over time, we calculated the difference between our adjusted “start” and “end” metadata, and the corresponding metadata from the Riksdag. The table below displays the total hours of audio per year, and the median of the difference between KBLab’s adjusted metadata and the Riksdag’s metadata. Negative <strong>start</strong> and <strong>end</strong> difference values reflect that KBLab’s speech starts or ends <em>earlier</em> in the debate audio file, whereas positive difference values reflects KBLab’s speech begins or ends <em>later</em> than the Riksdag’s.</p>
<div class="column-body-outset">
<div data-pagedtable="false">
<script data-pagedtable-source="" type="application/json">
{"columns":[{"label":["Year"],"name":[1],"type":["dbl"],"align":["right"]},{"label":["Total duration (hours)"],"name":[2],"type":["dbl"],"align":["right"]},{"label":["Median start difference (seconds)"],"name":[3],"type":["dbl"],"align":["right"]},{"label":["Median end difference (seconds)"],"name":[4],"type":["dbl"],"align":["right"]}],"data":[{"1":"2003","2":"7.86","3":"-2.58","4":"-14.00"},{"1":"2004","2":"74.37","3":"-3.83","4":"-16.72"},{"1":"2005","2":"35.79","3":"-9.43","4":"-29.17"},{"1":"2006","2":"185.79","3":"12.44","4":"-86.77"},{"1":"2007","2":"166.14","3":"12.22","4":"-85.44"},{"1":"2008","2":"266.62","3":"13.28","4":"-83.67"},{"1":"2009","2":"248.08","3":"13.92","4":"-83.93"},{"1":"2010","2":"222.14","3":"13.07","4":"-85.01"},{"1":"2011","2":"277.44","3":"11.00","4":"-82.78"},{"1":"2012","2":"294.33","3":"2.40","4":"-1.36"},{"1":"2013","2":"304.51","3":"3.04","4":"-1.28"},{"1":"2014","2":"209.41","3":"3.42","4":"-1.43"},{"1":"2015","2":"414.48","3":"2.93","4":"-1.38"},{"1":"2016","2":"438.99","3":"2.51","4":"-1.49"},{"1":"2017","2":"424.49","3":"2.57","4":"-1.63"},{"1":"2018","2":"343.28","3":"2.99","4":"-1.60"},{"1":"2019","2":"356.57","3":"2.92","4":"-1.75"},{"1":"2020","2":"354.47","3":"3.15","4":"-1.63"},{"1":"2021","2":"508.19","3":"3.43","4":"-1.78"},{"1":"2022","2":"472.72","3":"3.01","4":"-2.50"},{"1":"2023","2":"27.97","3":"0.18","4":"-5.43"}],"options":{"columns":{"min":{},"max":[4]},"rows":{"min":[8],"max":[8]},"pages":{}}}
  </script>
</div>
</div>
<p>We note how, for several years between 2006 and 2011, the adjusted KBLab metadata places the end of a speech about 85 seconds <em>earlier</em> than the Riksdag. The typical <strong>end</strong> indication from the Riksdag during these years appears to overshoot the speech by more than a minute. Looking at the same thing in graphical form below, where every point represents the mean difference within an entire debate, we see a systematic pattern of the “end” metadata marker from the Riksdag overshooting the end of speeches (Figure&nbsp;1 (b)). In contrast, most of the “start” metadata markings (Figure&nbsp;1 (a)) are more accurate. Perhaps this was a conscious choice, as capturing the start of a speech might have been deemed more important than spending a lot of time and resources capturing the exact ending of the speech.</p>
<div class="page-columns page-full">
<div id="fig-delta" class="quarto-layout-panel page-columns page-full">
<figure class="quarto-float quarto-float-fig figure page-columns page-full">
<div aria-describedby="fig-delta-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca" class="page-columns page-full">
<div class="quarto-layout-row column-page">
<div class="quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="fig-delta" style="flex-basis: 50.0%;justify-content: flex-start;">
<div id="fig-deltaleft" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-fig figure">
<div aria-describedby="fig-deltaleft-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="start_diff.jpg" class="lightbox" data-gallery="fig-delta" title="Figure&nbsp;1&nbsp;(a): The difference (delta) between ‘start’ time metadata for KBLab and the Riksdag"><img src="https://kb-labb.github.io/posts/2023-02-15-finding-speeches-in-the-riksdags-debates/start_diff.jpg" class="img-fluid figure-img" style="width:100.0%" data-ref-parent="fig-delta"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-subfloat-caption quarto-subfloat-fig" id="fig-deltaleft-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(a) The difference (delta) between ‘start’ time metadata for KBLab and the Riksdag
</figcaption>
</figure>
</div>
</div>
<div class="quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="fig-delta" style="flex-basis: 50.0%;justify-content: flex-start;">
<div id="fig-deltaright" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-fig figure">
<div aria-describedby="fig-deltaright-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="end_diff.jpg" class="lightbox" data-gallery="fig-delta" title="Figure&nbsp;1&nbsp;(b): The difference (delta) between ‘end’ time metadata for KBLab and the Riksdag"><img src="https://kb-labb.github.io/posts/2023-02-15-finding-speeches-in-the-riksdags-debates/end_diff.jpg" class="img-fluid figure-img" style="width:100.0%" data-ref-parent="fig-delta"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-subfloat-caption quarto-subfloat-fig" id="fig-deltaright-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(b) The difference (delta) between ‘end’ time metadata for KBLab and the Riksdag
</figcaption>
</figure>
</div>
</div>
</div>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-delta-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;1: Mean difference between KBLab adjusted metadata and the Riksdag. In general the Riksdag’s start markers tend to be biased towards beginning a few seconds before the actual speech, and ending later than the actual ending. The plot shows a zoomed in view between -200 and +200 seconds to emphasize general trends. There also exist debates for which the metadata is off by thousands of seconds.
</figcaption>
</figure>
</div>
</div>
</section>
<section id="most-and-least-intelligible-speakers" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="most-and-least-intelligible-speakers">Most and least intelligible speakers</h3>
<p>After adjusting the metadata, we used the new metadata to split the debate audio file in to speech segments. These speech files were machine transcribed and evaluated against the official transcript using the <a href="https://en.wikipedia.org/wiki/BLEU">BLEU</a> score. High BLEU score generally indicate there’s a high correspondence or overlap between the output of the machine transcription and the official transcript.</p>
<p>In the table below, we display the 30 speakers with the lowest mean BLEU score, among those speakers with more than <img src="https://latex.codecogs.com/png.latex?5"> speeches in the Riksdag’s debates. A low BLEU score may not necessarily imply the speaker is less intelligible for speech-to-text models, but may be the result of a combination of different factors:</p>
<ol type="1">
<li>The official transcription may have taken less or more liberties when transcribing the speech.</li>
<li>The segmentation (adjusted metadata) may have been systematically off for a particular speaker.</li>
</ol>
<p>Interestingly, the vast majority of the bottom <img src="https://latex.codecogs.com/png.latex?30"> list are men (<img src="https://latex.codecogs.com/png.latex?28"> out of <img src="https://latex.codecogs.com/png.latex?30">). Many of the members of parliament on this list have thick accents, or speak in dialectal varieties of Swedish. The low BLEU scores either indicate that speech-to-text models perform worse for this subset, or that the official transcriptions tend to rephrase a lot of what was uttered.</p>
<div class="column-body-outset">
<div data-pagedtable="false">
<script data-pagedtable-source="" type="application/json">
{"columns":[{"label":["Speaker"],"name":[1],"type":["chr"],"align":["left"]},{"label":["Party"],"name":[2],"type":["chr"],"align":["left"]},{"label":["Gender"],"name":[3],"type":["chr"],"align":["left"]},{"label":["Electoral District"],"name":[4],"type":["chr"],"align":["left"]},{"label":["Mean BLEU Score"],"name":[5],"type":["dbl"],"align":["right"]}],"data":[{"1":"Jasenko Omanovic","2":"S","3":"male","4":"Västernorrlands län","5":"0.3150381"},{"1":"Momodou Malcolm Jallow","2":"V","3":"NA","4":"NA","5":"0.3202471"},{"1":"Mikael Dahlqvist","2":"S","3":"male","4":"Värmlands län","5":"0.3220909"},{"1":"Nikos Papadopoulos","2":"S","3":"male","4":"Stockholms kommun","5":"0.3246828"},{"1":"Roger Richthoff","2":"-","3":"male","4":"Södermanlands län","5":"0.3339884"},{"1":"Lars Thomsson","2":"C","3":"male","4":"Gotlands län","5":"0.3383261"},{"1":"Amineh Kakabaveh","2":"-","3":"female","4":"Stockholms län","5":"0.3395919"},{"1":"Göran S Persson","2":"S","3":"male","4":"Skåne läns norra och östra","5":"0.3399949"},{"1":"Lars Tysklind","2":"L","3":"male","4":"Västra Götalands läns västra","5":"0.3401489"},{"1":"Ronny Olander","2":"S","3":"male","4":"Skåne läns södra","5":"0.3537719"},{"1":"Lars U Granberg","2":"S","3":"male","4":"Norrbottens län","5":"0.3601472"},{"1":"Peter Jonsson","2":"S","3":"NA","4":"NA","5":"0.3637517"},{"1":"Malcolm Momodou Jallow","2":"V","3":"male","4":"Malmö kommun","5":"0.3681529"},{"1":"Magnus Ehrencrona","2":"MP","3":"male","4":"Skåne läns västra","5":"0.3751171"},{"1":"Magnus Oscarsson","2":"KD","3":"male","4":"Östergötlands län","5":"0.3758436"},{"1":"Jamal Mouneimne","2":"S","3":"male","4":"Västerbottens län","5":"0.3761942"},{"1":"Dag Klackenberg","2":"M","3":"male","4":"Stockholms kommun","5":"0.3836031"},{"1":"Arhe Hamednaca","2":"S","3":"male","4":"Stockholms kommun","5":"0.3840110"},{"1":"Peter Pedersen","2":"V","3":"male","4":"Örebro län","5":"0.3856092"},{"1":"Rolf Gunnarsson","2":"M","3":"male","4":"Dalarnas län","5":"0.3940639"},{"1":"Otto von Arnold","2":"KD","3":"male","4":"Skåne läns södra","5":"0.3964348"},{"1":"Sotiris Delis","2":"M","3":"male","4":"Jönköpings län","5":"0.3970086"},{"1":"Mauricio Rojas","2":"L","3":"male","4":"Skåne läns västra","5":"0.3976366"},{"1":"Marie-Louise Hänel Sandström","2":"M","3":"female","4":"Göteborgs kommun","5":"0.3980042"},{"1":"Mikael Oscarsson","2":"KD","3":"male","4":"Uppsala län","5":"0.3993218"},{"1":"Hans Olsson","2":"S","3":"male","4":"Västra Götalands läns södra","5":"0.4025808"},{"1":"Jamal El-Haj","2":"S","3":"male","4":"Malmö kommun","5":"0.4025900"},{"1":"Rezene Tesfazion","2":"S","3":"male","4":"Uppsala län","5":"0.4072008"},{"1":"Torbjörn Björlund","2":"V","3":"male","4":"Östergötlands län","5":"0.4080772"},{"1":"Lars Mejern Larsson","2":"S","3":"male","4":"Värmlands län","5":"0.4122340"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}}
  </script>
</div>
</div>
<p>The top <img src="https://latex.codecogs.com/png.latex?30"> list, in contrast, seems to skew towards high BLEU scores for women (<img src="https://latex.codecogs.com/png.latex?21"> out of <img src="https://latex.codecogs.com/png.latex?30">). An interesting research question to examine would be whether this to a higher degree can be attributed to the speech-to-text model, how transcriptions are written, or the degree to which speeches are prepared or improvised.</p>
<div class="column-body-outset">
<div data-pagedtable="false">
<script data-pagedtable-source="" type="application/json">
{"columns":[{"label":["Speaker"],"name":[1],"type":["chr"],"align":["left"]},{"label":["Party"],"name":[2],"type":["chr"],"align":["left"]},{"label":["Gender"],"name":[3],"type":["chr"],"align":["left"]},{"label":["Electoral District"],"name":[4],"type":["chr"],"align":["left"]},{"label":["Mean BLEU Score"],"name":[5],"type":["dbl"],"align":["right"]}],"data":[{"1":"Andrea Andersson Tay","2":"V","3":"female","4":"Stockholms län","5":"0.7919546"},{"1":"Christina Tapper Östberg","2":"SD","3":"female","4":"Skåne läns norra och östra","5":"0.7720259"},{"1":"Cassandra Sundin","2":"SD","3":"female","4":"Jämtlands län","5":"0.7621053"},{"1":"Karl Sigfrid","2":"M","3":"male","4":"Stockholms län","5":"0.7610473"},{"1":"Helena Antoni","2":"M","3":"NA","4":"NA","5":"0.7482292"},{"1":"Jacob Risberg","2":"MP","3":"male","4":"Uppsala län","5":"0.7403408"},{"1":"Pia Hallström","2":"M","3":"female","4":"Värmlands län","5":"0.7399343"},{"1":"Aida Birinxhiku","2":"S","3":"female","4":"Hallands län","5":"0.7351579"},{"1":"Anna Sibinska","2":"MP","3":"female","4":"Göteborgs kommun","5":"0.7295161"},{"1":"Torsten Lindström","2":"KD","3":"male","4":"Västmanlands län","5":"0.7224670"},{"1":"Anna Vikström","2":"S","3":"female","4":"Stockholms län","5":"0.7171087"},{"1":"Angelica Lundberg","2":"SD","3":"female","4":"Västmanlands län","5":"0.7154092"},{"1":"Kristoffer Lindberg","2":"S","3":"male","4":"Gävleborgs län","5":"0.7142424"},{"1":"Jessica Rodén","2":"S","3":"female","4":"Västra Götalands läns södra","5":"0.7133541"},{"1":"Kajsa Lunderquist","2":"M","3":"female","4":"Malmö kommun","5":"0.7125567"},{"1":"Jessica Thunander","2":"V","3":"female","4":"Västra Götalands läns östra","5":"0.7124428"},{"1":"Jeanette Gustafsdotter","2":"S","3":"female","4":"NA","5":"0.7086785"},{"1":"Carina Moberg","2":"S","3":"female","4":"Stockholms län","5":"0.7054473"},{"1":"Lars Jilmstad","2":"M","3":"male","4":"Stockholms kommun","5":"0.7052950"},{"1":"Clara Aranda","2":"SD","3":"female","4":"Östergötlands län","5":"0.7045666"},{"1":"Lena Sommestad","2":"S","3":"female","4":"Uppsala län","5":"0.7044653"},{"1":"Hampus Hagman","2":"KD","3":"male","4":"Göteborgs kommun","5":"0.7029829"},{"1":"Sofia Modigh","2":"KD","3":"female","4":"Stockholms kommun","5":"0.7024092"},{"1":"Jenny Bengtsson","2":"V","3":"female","4":"Stockholms län","5":"0.7020539"},{"1":"Linda Modig","2":"C","3":"female","4":"Norrbottens län","5":"0.6994925"},{"1":"Inger Fredriksson","2":"C","3":"female","4":"Södermanlands län","5":"0.6980692"},{"1":"Leila Ali Elmi","2":"MP","3":"female","4":"Göteborgs kommun","5":"0.6964871"},{"1":"Lars Isovaara","2":"SD","3":"male","4":"Uppsala län","5":"0.6955071"},{"1":"Fredrik Stenberg","2":"S","3":"male","4":"Västerbottens län","5":"0.6953429"},{"1":"Mikael Damsgaard","2":"M","3":"male","4":"Västmanlands län","5":"0.6952271"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}}
  </script>
</div>
</div>
<p>You can use the search feature on the Riksdag’s Web TV to <a href="https://www.riksdagen.se/sv/webb-tv/?q=andrea+andersson+tay&amp;p=1&amp;st=3">search for debates these speakers participated in</a>.</p>
</section>
</section>
<section id="the-speech-finder-method" class="level2">
<h2 class="anchored" data-anchor-id="the-speech-finder-method">The speech finder method</h2>
<p>We’ll provide a short summary of our method here, since the concept is likely better illustrated in graphical form.</p>
<ol type="1">
<li>We begin by using a speech-to-text model to transcribe an entire audio file. The speech-to-text model also outputs timestamps for when each of its output words were uttered. We use the model wav2vec2-large-voxrex-swedish <span class="citation" data-cites="wav2vec2">(Malmsten, Haffenden, and Börjeson 2022)</span>.</li>
<li>We use the official transcripts, and perform fuzzy string matching to locate what region of the automatic transcript they match against. The string matched region becomes an approximation of where the speech is located in the big audio file.</li>
<li>We perform <a href="https://en.wikipedia.org/wiki/Speaker_diarisation">speaker diarization</a>. This gives us a more detailed view of when different speakers spoke during the debate. We use the tool pyannote.audio <span class="citation" data-cites="Bredin2020">(Bredin et al. 2020)</span> <span class="citation" data-cites="Bredin2021">(Bredin and Laurent 2021)</span> for speaker diarization.</li>
<li>We now still need to connect the speaker segments from the speaker diarization output to the official transcripts and the metadata associated with the transcripts. We therefore use the approximate predictions from the string matching method, and calculate the degre of overlap between different diarization speaker segments and the fuzzy string matched region of a speech.</li>
<li>The dominant (most overlapping) speaker segment(s) becomes our prediction.</li>
</ol>
<p>The idea is that this method should be completely independent way to generate speech segments – one that does not rely on the Riksdag’s already existing metadata. The only necessary components are i) a transcript, and ii) an audio file. The objective is essentially to find the needle (speech) in the haystack (audio file).</p>
<p><a href="speech_finder_method.png" class="lightbox" data-gallery="quarto-lightbox-gallery-4"><img src="https://kb-labb.github.io/posts/2023-02-15-finding-speeches-in-the-riksdags-debates/speech_finder_method.png" class="img-fluid" style="width:100.0%" alt="An illustrative sketch with text, describing in 5 steps how KBLab's method for finding speeches in audio files works. Step 1 is to use automatic speech recognition to transcribe an audio file. Step 2 is to fuzzy string match the ASR output against official transcripts to get approximate start and end locations for a speech. Step 3 is to perform speaker diarization to partition the audio file in to segments of different speakers. Step 4 is to assign speaker diarization segments with high degree of overlap with the speeches associated with the approximate start and end locations as predicted by fuzzy string match. Step 5 is to use start and end locations of the assigned segments as the new predictions of metadata for when a speech begins and ends."></a></p>
</section>
<section id="evaluation-of-metadata-quality" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="evaluation-of-metadata-quality">Evaluation of metadata quality</h2>
<p>How do we know whether our method works or not? The benefit of the “speech finder” method being independent of the Riksdag’s official metadata, is that we can benchmark and compare against the portions of Riksdag data that maintain the best metadata quality. In general the best metadata quality is found in recent years, although most years since 2012 have had a fairly consistent and high quality.</p>
<p>In the plot below, we plot the average monthly BLEU scores over time (higher is better). KBLab’s adjusted metadata is of comparable quality the Riksdag’s metadata for, 2012 and forward. However, whereas the Riksdag’s BLEU scores drop for older debates, speech segmentations based on KBLabs adjusted metadata maintains an even and similar quality throughout the entire period.</p>
<div class="page-columns page-full">
<div id="fig-bleu" class="quarto-float quarto-figure quarto-figure-center anchored page-columns page-full">
<figure class="quarto-float quarto-float-fig figure page-columns page-full">
<div aria-describedby="fig-bleu-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca" class="page-columns page-full">
<a href="monthly_bleu.jpg" class="lightbox page-columns page-full" data-gallery="quarto-lightbox-gallery-5" title="Figure&nbsp;2: BLEU score is often used as a scoring metric for determining how well a translation overlaps with a reference, source or ground truth text. In our case we split the debate audio file in to speeches based on ‘start’ and ‘end’ metadata. We then use the wav2vec2-voxrex-swedish speech-to-text model to automatically transcribe any speech within the split speeches. The automatic model transcription can be seen as a form of ‘translation’. Finally, we compare the automated transcription against the official transcript. A high BLEU score indicates what was spoken within the segmented region overlaps to a high degree with the official transcript."><img src="https://kb-labb.github.io/posts/2023-02-15-finding-speeches-in-the-riksdags-debates/monthly_bleu.jpg" class="img-fluid figure-img column-page-inset" style="width:100.0%"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-bleu-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;2: BLEU score is often used as a scoring metric for determining how well a translation overlaps with a reference, source or ground truth text. In our case we split the debate audio file in to speeches based on ‘start’ and ‘end’ metadata. We then use the wav2vec2-voxrex-swedish speech-to-text model to automatically transcribe any speech within the split speeches. The automatic model transcription can be seen as a form of ‘translation’. Finally, we compare the automated transcription against the official transcript. A high BLEU score indicates what was spoken within the segmented region overlaps to a high degree with the official transcript.
</figcaption>
</figure>
</div>
</div>
</section>
<section id="the-benefits-of-open-data" class="level2">
<h2 class="anchored" data-anchor-id="the-benefits-of-open-data">The benefits of open data</h2>
<p>We like open data at KBLab! The Riksdag’s open data is a wonderful resource that we believe will benefit research for many years to come. Making it easy for many people to acccess and use the Riksdag’s data creates positive spinoff effects. The <a href="https://www.westac.se/en/">WeStAc</a> project with their curation of protocols from the Riksdag spanning back to the 1920s is one such example (see <a href="https://github.com/welfare-state-analytics/riksdagen-corpus">riksdagen-corpus</a>). While KBLab’s primary objective in doing speech segmentation was to create an ASR dataset, the method and metadata output can be shared and used by both the Riksdag and <a href="https://www.vr.se/swecris.html#/project/IN22-0003_RJ">other projects</a> interested in curating Riksdag materials and possibly connecting curated protocols to audio and video recordings.</p>
<p>The audiovisual materials currently exposed in the Riksdag’s APIs has made it possible for us to create RixVox: a 5500 hour audio dataset with aligned text transcripts. From the point of view of KBLab, we benefit greatly from other organizations and research projects curating datasets, protocols and transcripts in this manner. We believe the method described in this article can likely successfully be applied to Riksdag debates from the 1960s to the 2000s. The prerequisite is simply having access to the audio, and a set of curated and well segmented textual transcripts from protocols. Luckily, since the Riksdag has done such a splendid job with their open data platform, there are research projects that have taken on the challenge!</p>
</section>
<section id="code-and-data" class="level2">
<h2 class="anchored" data-anchor-id="code-and-data">Code and data</h2>
<p>We plan on making RixVox, the 5500 hour speech-to-text dataset consisting of parliamentary speeches, freely and openly available on Huggingface. A new post will appear on this blog announcing the news when it is released!</p>
<p>The code for reproducing the speech segmentations, the adjusted metadata, and RixVox is available. See the <strong>Code</strong> and <strong>Data</strong> sections below.</p>
</section>
<section id="acknowledgements" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="acknowledgements">Acknowledgements</h2>
<p>Part of this development work was carried out within the frame of the infrastructural project <a href="https://www.huminfra.se/">HUMINFRA</a>.</p>

<div class="no-row-height column-margin column-container"><div class="">
<div class="quarto-figure quarto-figure-left">
<figure class="figure">
<p><a href="huminfra.svg" class="lightbox" data-gallery="quarto-lightbox-gallery-6"><img src="https://kb-labb.github.io/posts/2023-02-15-finding-speeches-in-the-riksdags-debates/huminfra.svg" class="img-fluid quarto-figure quarto-figure-left figure-img" style="width:35.0%"></a></p>
</figure>
</div>
</div></div></section>




<div id="quarto-appendix" class="default"><section id="code" class="level2 appendix"><h2 class="anchored quarto-appendix-heading">Code</h2><div class="quarto-appendix-contents">

<p>The code for reproducing results in this article can be found on https://github.com/kb-labb/riksdagen_anforanden.</p>
</div></section><section id="data" class="level2 appendix"><h2 class="anchored quarto-appendix-heading">Data</h2><div class="quarto-appendix-contents">

<p>The resulting metadata can be downloaded, and has been shared with the Riksdag. You can find the metadata here: https://github.com/kb-labb/riksdagen_anforanden/tree/main/metadata.</p>
<div class="cell">
<script>
var metadata = data_example;

var i = 0;

transcript1 = document.getElementById("speechtranscript");
transcript2 = document.getElementById("speechtranscript2");
metadata1 = document.getElementById("metadatastart");
metadata2 = document.getElementById("metadatastart2");
speaker1 = document.getElementById("speaker1");
speaker2 = document.getElementById("speaker2");

// Initialize transcripts
transcript1.textContent = metadata[i].anftext_short;
transcript2.textContent = metadata[i].anftext_short;

//add event listener
prevspeech.addEventListener('click', prevClickEvent);
prevspeech2.addEventListener('click', prevClickEvent);
nextspeech.addEventListener('click', nextClickEvent);
nextspeech2.addEventListener('click', nextClickEvent);

function embedLinkGetter(debateurl, start, end){
  base_url = "https://www.riksdagen.se";
  embed_url = base_url + debateurl + "/embed/?start=" + start + "&end=" + end;
  return embed_url
}

function changeVideo(video1_id, video2_id, is_debate=false){
  if (is_debate) {
    document.getElementById(video1_id).src = embedLinkGetter(metadata_debate[j].debateurl, metadata_debate[j].start_adjusted, metadata_debate[j].end_adjusted);
    document.getElementById(video2_id).src = embedLinkGetter(metadata_debate[j].debateurl, metadata_debate[j].start, metadata_debate[j].end);
  } else {
    document.getElementById(video1_id).src = embedLinkGetter(metadata[i].debateurl, metadata[i].start_adjusted, metadata[i].end_adjusted);
    document.getElementById(video2_id).src = embedLinkGetter(metadata[i].debateurl, metadata[i].start, metadata[i].end);
  }
}

function secondsToHourMinSec(seconds, begin=11, end=21){
  return new Date(seconds * 1000).toISOString().slice(begin, end);
}

function editMetadata(i, is_debate=false){
  if (is_debate){
    console.log("is debate here");
    // The orange/yellow boxes with debate buttons
    transcript3.textContent = metadata_debate[i].anftext_short;
    transcript4.textContent = metadata_debate[i].anftext_short;
    metadata3.innerHTML = "<b>Start:</b> " + secondsToHourMinSec(metadata_debate[i].start_adjusted);
    metadata3.innerHTML += "<b>  End:</b> " + secondsToHourMinSec(metadata_debate[i].end_adjusted);
    metadata4.innerHTML = "<b>Start:</b> " + secondsToHourMinSec(metadata_debate[i].start);
    metadata4.innerHTML += "<b>  End:</b> " + secondsToHourMinSec(metadata_debate[i].end); 
    speaker3.innerHTML = "<b>Speaker: </b> " + metadata_debate[i].speaker + " (" + metadata_debate[i].party + ")";
    speaker4.innerHTML = "<b>Speaker: </b> " + metadata_debate[i].speaker_audio_meta; 
  } else {
    transcript1.textContent = metadata[i].anftext_short;
    transcript2.textContent = metadata[i].anftext_short;
    metadata1.innerHTML = "<b>Start:</b> " + secondsToHourMinSec(metadata[i].start_adjusted);
    metadata1.innerHTML += "<b>  End:</b> " + secondsToHourMinSec(metadata[i].end_adjusted);
    metadata2.innerHTML = "<b>Start:</b> " + secondsToHourMinSec(metadata[i].start);
    metadata2.innerHTML += "<b>  End:</b> " + secondsToHourMinSec(metadata[i].end); 
    speaker1.innerHTML = "<b>Speaker: </b> " + metadata[i].speaker + " (" + metadata[i].party + ")";
    speaker2.innerHTML = "<b>Speaker: </b> " + metadata[i].speaker_audio_meta; 
  }
}

function prevClickEvent() {
  if (i > 0) {
    i--;
    changeVideo("videokb", "videoriks", is_debate=false);
  }

  editMetadata(i);
}

function nextClickEvent() {
  if (i < (metadata.length - 1)) {
    i++;
    changeVideo("videokb", "videoriks", is_debate=false);
  }
  
  editMetadata(i);
}
</script>
</div>
<div class="cell">
<script>
var j = 0;
var k = 0;

debate_ids = Object.keys(data_debate);
metadata_debate = data_debate[debate_ids[k]];
transcript3 = document.getElementById("speechtranscript3");
transcript4 = document.getElementById("speechtranscript4");
metadata3 = document.getElementById("metadatastart3");
metadata4 = document.getElementById("metadatastart4");
speaker3 = document.getElementById("speaker3");
speaker4 = document.getElementById("speaker4");

// Initialize transcripts
transcript3.textContent = metadata_debate[j].anftext_short;
transcript4.textContent = metadata_debate[j].anftext_short;

//add event listener
prevspeech3.addEventListener('click', prevSpeechEvent);
prevspeech4.addEventListener('click', prevSpeechEvent);
nextspeech3.addEventListener('click', nextSpeechEvent);
nextspeech4.addEventListener('click', nextSpeechEvent);
prevdebate1.addEventListener('click', prevDebateEvent);
prevdebate2.addEventListener('click', prevDebateEvent);
nextdebate1.addEventListener('click', nextDebateEvent);
nextdebate2.addEventListener('click', nextDebateEvent);

function embedLinkGetter(debateurl, start, end){
  // debateurl example: /sv/webb-tv/video/fragestund/fragestund_H9C120220512fs
  base_url = "https://www.riksdagen.se";
  embed_url = base_url + debateurl + "/embed/?start=" + start + "&end=" + end;
  return embed_url
}

function prevSpeechEvent() {
  if (j > 0) {
    j--;
    changeVideo("videokb_deb", "videoriks_deb", is_debate=true);
  }

  editMetadata(j, is_debate=true);
}

function nextSpeechEvent() {
  if (j < (metadata.length - 1)) {
    j++;
    changeVideo("videokb_deb", "videoriks_deb", is_debate=true);
  }
  
  editMetadata(j, is_debate=true);
}

function prevDebateEvent() {
  if (k > 0){
    k--;
    j = 0;
    metadata_debate = data_debate[debate_ids[k]];
    changeVideo("videokb_deb", "videoriks_deb", is_debate=true);
  }
  
  editMetadata(j, is_debate=true);
}

function nextDebateEvent() {
  if (k < (debate_ids.length - 1)){
    k++;
    j = 0;
    metadata_debate = data_debate[debate_ids[k]];
    changeVideo("videokb_deb", "videoriks_deb", is_debate=true);
  }
  
  editMetadata(j, is_debate=true);
}
</script>
</div>
<div class="cell">
<style type="text/css">
.videobox-container {
  width: 1060px; /* Adjust this width to fit both videoboxes + margin */
  margin: 0 auto; /* Center the container */
  overflow: hidden; /* Clear floats */
}

.videobox{
    float:left;
    align-content: center;
    margin-right:20px;
    border:2px;
    word-wrap: break-word;
    max-width: 510px;
}

.clear{
    clear:both;
}

.metabox{
    background-color: #d1d1e0;
    margin: 0px;
}

.metabox > * {
  padding-left: 6px;
}

.metabox_debate{
    background-color: #ffdab3;
    margin: 0px;
}

.metabox_debate > * {
  padding-left: 6px;
}

.beginning {
  margin-bottom: 4px;
}
</style>
</div>



</div></section><section class="quarto-appendix-contents" id="quarto-bibliography"><h2 class="anchored quarto-appendix-heading">References</h2><div id="refs" class="references csl-bib-body hanging-indent" data-entry-spacing="0">
<div id="ref-Bredin2021" class="csl-entry">
Bredin, Hervé, and Antoine Laurent. 2021. <span>“<span class="nocase">End-to-end speaker segmentation for overlap-aware resegmentation</span>.”</span> In <em>Proc. Interspeech 2021</em>.
</div>
<div id="ref-Bredin2020" class="csl-entry">
Bredin, Hervé, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, and Marie-Philippe Gill. 2020. <span>“<span class="nocase">pyannote.audio: neural building blocks for speaker diarization</span>.”</span> In <em>ICASSP 2020, IEEE International Conference on Acoustics, Speech, and Signal Processing</em>.
</div>
<div id="ref-wav2vec2" class="csl-entry">
Malmsten, Martin, Chris Haffenden, and Love Börjeson. 2022. <span>“Hearing Voices at the National Library – a Speech Corpus and Acoustic Model for the Swedish Language.”</span> <a href="https://arxiv.org/abs/2205.03026">https://arxiv.org/abs/2205.03026</a>.
</div>
</div></section><section id="footnotes" class="footnotes footnotes-end-of-document"><h2 class="anchored quarto-appendix-heading">Footnotes</h2>

<ol>
<li id="fn1"><p>An example can be seen here: <a href="https://www.riksdagen.se/sv/webb-tv/video/partiledardebatt/partiledardebatt_HAC120230118pd">https://www.riksdagen.se/sv/webb-tv/video/partiledardebatt/partiledardebatt_HAC120230118pd</a>↩︎</p></li>
<li id="fn2"><p>The file Sagtochgjort.csv.zip here: <a href="https://data.riksdagen.se/data/ledamoter/">https://data.riksdagen.se/data/ledamoter/</a>↩︎</p></li>
</ol>
</section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{rekathati2023,
  author = {Rekathati, Faton},
  title = {Finding {Speeches} in the {Riksdag’s} {Debates}},
  date = {2023-02-23},
  url = {https://kb-labb.github.io/posts/2023-02-15-finding-speeches-in-the-riksdags-debates/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-rekathati2023" class="csl-entry quarto-appendix-citeas">
Rekathati, Faton. 2023. <span>“Finding Speeches in the Riksdag’s
Debates.”</span> February 23, 2023. <a href="https://kb-labb.github.io/posts/2023-02-15-finding-speeches-in-the-riksdags-debates/">https://kb-labb.github.io/posts/2023-02-15-finding-speeches-in-the-riksdags-debates/</a>.
</div></div></section></div> ]]></description>
  <guid>https://kb-labb.github.io/posts/2023-02-15-finding-speeches-in-the-riksdags-debates/</guid>
  <pubDate>Wed, 22 Feb 2023 23:00:00 GMT</pubDate>
</item>
<item>
  <title>Swedish zero-shot classification model</title>
  <dc:creator>Justyna Sikora</dc:creator>
  <link>https://kb-labb.github.io/posts/2023-02-12-zero-shot-text-classification/</link>
  <description><![CDATA[ 





<section id="what-is-zero-shot-text-classification" class="level2">
<h2 class="anchored" data-anchor-id="what-is-zero-shot-text-classification">What is zero-shot text classification?</h2>
<p>The general goal of text classification is to assign a label to a piece of writing – whether it is a book review that we want to categorize as positive or negative, or a news article that we wish to check the topic of. Use cases are diverse and can range from topic classification and spam detection to sentiment analysis.</p>
<p>Usually, text classification is conducted with help of supervised machine learning, which means that we have to gather a sufficient number of labeled examples in order to train a classification model. The model learns relations between the sentences and the target labels and in turn can extrapolate from the examples it was exposed to during training to the unseen data. For instance, if we want to use a sentiment classification model, we first have to feed it with a train portion of the data, which can contain movie or book reviews marked as negative, positive or neutral. Only after training, we can show new texts to the model and ask it to predict the sentiment of the previously unseen reviews.</p>
<p>The procedure is straightforward, however, one issue is that it requires a labeled dataset for training. If we don’t have one, or cannot create one, generating the model is impossible. A solution suggested by Yin et al.&nbsp;in “Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach” (2019) has its roots in changing perspective and treating classification task as a textual entailment problem. This approach allows us to use already available Natural Language Inference (NLI) models as zero-shot text classifiers. In zero-shot learning, models are not exposed to any task-specific examples, yet they are expected to generalize from the tasks they were trained on to others – in our case NLI models and various classification tasks.</p>
</section>
<section id="how-can-natural-language-inference-models-be-used-with-zero-shot-classification" class="level2">
<h2 class="anchored" data-anchor-id="how-can-natural-language-inference-models-be-used-with-zero-shot-classification">How can Natural Language Inference models be used with zero-shot classification?</h2>
<p>Natural Language Inference task is designed to test models’ reasoning capabilities. The aim of the task is to decide if there is a connection between two sentences, so-called premise and hypothesis, by checking whether a premise entails, contradicts or is not related to a hypothesis. For instance, the premise sentence “How do you know? All this is their information again.” entails the hypothesis “This information belongs to them.”, while the premise “but that takes too much planning” is contradictory to the hypothesis “It doesn’t take much planning.”. The idea behind using NLI models as zero-shot text classifiers is to look at input text that we want to categorize and its label as an entailment pair. If the premise entails the hypothesis, we can assume that the label is suitable for the text, otherwise we reject it.</p>
<p>In order to be able to evaluate entailment pairs, we first have to reformulate potential classes into hypotheses with help of a template. Templates can be adjusted to specific tasks - for topic classification we could use the templates “This text is about {}.” or “This example is {}.”, while “This text expresses {}.” might be appropriate for sentiment analysis.</p>
<p>The pairs can be constructed in the following manner:</p>
<p><strong>Premise</strong>: “När Tutankhamons grav upptäcktes för 100 år sedan blev han världskänd. Faraon dog som 19-åring för över 3000 år sedan men när graven hittades var den helt intakt. Forskarna trodde först att Tutankhamon blivit mördad, men egyptologen Zahi Hawass tror sig vara nära att knäcka gåtan om den unga faraons död.”</p>
<p><strong>Categories</strong>: “sport”, “religion”, “nöje”, “politik”, “vetenskap”</p>
<p><strong>Template</strong>: “Detta exempel handlar om {}.”</p>
<p><strong>First hypothesis</strong>: “Detta exempel handlar om sport.”</p>
<p>By inserting the labels into the template and evaluating each of the pairs, we can check which topic is most suitable for the premise.</p>
</section>
<section id="training-the-swedish-model" class="level2">
<h2 class="anchored" data-anchor-id="training-the-swedish-model">Training the Swedish model</h2>
<p>To create the Swedish model for zero-shot text classification, we have experimented with fine-tuning models on different combinations of NLI tasks. From the models manually tested on a set of entailment pairs, the best results obtained Swedish BERT-large model with 340M parameters fine-tuned on two NLI datasets, which are part of the Swedish version of the GLUE benchmark (https://gluebenchmark.com/) – OverLim (https://huggingface.co/datasets/KBLab/overlim). The fine-tuning was conducted in two steps, first on The Stanford Question Answering Dataset (QNLI) and then on The Multi-Genre Natural Language Inference Corpus (MNLI). The model obtained 91.23% and 84.71% accuracy respectively.</p>
</section>
<section id="examples" class="level2">
<h2 class="anchored" data-anchor-id="examples">Examples</h2>
<p>The model is available on the KBLab’s HuggingFace page and can be used with 🤗 zero-shot classification pipeline. The pipeline offers a convenient API for using NLI models for classification. Let’s have a look at some examples!</p>
<p><a href="import.png" class="lightbox" data-gallery="quarto-lightbox-gallery-1"><img src="https://kb-labb.github.io/posts/2023-02-12-zero-shot-text-classification/import.png" class="img-fluid"></a></p>
<p>After loading the model, we have to specify a premise sentence and a list of categories that we wish to evaluate together with the input text. The output is a list over classes and corresponding probabilities calculated from probabilities for entailment and contradiction.</p>
<p><a href="output1.png" class="lightbox" data-gallery="quarto-lightbox-gallery-2"><img src="https://kb-labb.github.io/posts/2023-02-12-zero-shot-text-classification/output1.png" class="img-fluid"></a></p>
<p>Defining a hypothesis template is not obligatory, however, setting a custom one can positively influence performance. In the example below, we can observe that the probability for the label “nöje” (entertainment) has increased when the template was changed to a more specific one.</p>
<p><a href="output2.png" class="lightbox" data-gallery="quarto-lightbox-gallery-3"><img src="https://kb-labb.github.io/posts/2023-02-12-zero-shot-text-classification/output2.png" class="img-fluid"></a></p>
<p>Let’s consider another example. Once the punctuation is removed, the probability decreases.</p>
<p><a href="output3.png" class="lightbox" data-gallery="quarto-lightbox-gallery-4"><img src="https://kb-labb.github.io/posts/2023-02-12-zero-shot-text-classification/output3.png" class="img-fluid"></a></p>
<p>With zero-shot approach training or fine-tuning models for text classification is not necessary. It can be therefore convenient if we are not in possession of labeled data, as the already trained NLI models can be used off-the-shelf. In addition, since the categories are not solely limited to ones determined at the point of training of a model, using a NLI-based zero-shot text classification pipeline gives more flexibility in terms of choice of candidate labels.</p>
<section id="links-about-zero-shot-text-classification" class="level3">
<h3 class="anchored" data-anchor-id="links-about-zero-shot-text-classification">Links about zero-shot text classification:</h3>
<p>“Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach”, Yin et al.&nbsp;article : https://aclanthology.org/D19-1404/</p>
<p>NLI model for Swedish: https://huggingface.co/KBLab/megatron-bert-large-swedish-cased-165-zero-shot</p>
<p>Information about HuggingFace pipelines: https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/pipelines</p>


</section>
</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{sikora2023,
  author = {Sikora, Justyna},
  title = {Swedish Zero-Shot Classification Model},
  date = {2023-02-13},
  url = {https://kb-labb.github.io/posts/2023-02-12-zero-shot-text-classification/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-sikora2023" class="csl-entry quarto-appendix-citeas">
Sikora, Justyna. 2023. <span>“Swedish Zero-Shot Classification
Model.”</span> February 13, 2023. <a href="https://kb-labb.github.io/posts/2023-02-12-zero-shot-text-classification/">https://kb-labb.github.io/posts/2023-02-12-zero-shot-text-classification/</a>.
</div></div></section></div> ]]></description>
  <guid>https://kb-labb.github.io/posts/2023-02-12-zero-shot-text-classification/</guid>
  <pubDate>Sun, 12 Feb 2023 23:00:00 GMT</pubDate>
  <media:content url="https://kb-labb.github.io/posts/2023-02-12-zero-shot-text-classification/cover.png" medium="image" type="image/png" height="89" width="144"/>
</item>
<item>
  <title>Swedish Sentence Transformer 2.0</title>
  <dc:creator>Faton Rekathati</dc:creator>
  <link>https://kb-labb.github.io/posts/2023-01-16-sentence-transformer-20/</link>
  <description><![CDATA[ 





<p>We release an updated Swedish sentence transformer model. In addition to training the model on parallel sentences, we concatenate sentences from those parallel text corpora whose sentence orderings are sequential, training our model on longer text paragraphs. The new <strong>KB-SBERT v2.0</strong> has an increased maximum sequence length of 384, up from the 256 maximum tokens of the previous model. The model performs only marginally worse on SuperLim’s SweParaphrase benchmark, while performing significantly better on SweFaQ.</p>
<section id="the-sequence-length-problem" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="the-sequence-length-problem">The sequence length problem</h2>
<p>The training of the vast majority of available non-English sentence transformer models involves the use of so called <em>parallel corpora</em>. These are text translations of the same source documents in two or more languages. Typically these datasets come in the form of sentence-aligned observations. As a result, the context of any single given observation is quite small.</p>

<div class="no-row-height column-margin column-container"><div class="">
<p>You can read more about the first version of the model in another article on our blog: <a href="https://kb-labb.github.io/posts/2021-08-23-a-swedish-sentence-transformer/">Introducing a Swedish Sentence Transformer</a>.</p>
</div></div><p>The setup for training the first version of KB-SBERT involved parallel corpora and using <em>knowledge distillation</em> to transfer the knowledge of an English sentence transformer to a Swedish student BERT model. The library <a href="https://github.com/UKPLab/sentence-transformers"><code>sentence-transformers</code></a> provides a template for training models in this manner. Its default setting however recommends a maximum sequence length of <img src="https://latex.codecogs.com/png.latex?128"> for knowledge distilled bi- or multilingual models. This is due to most training examples in parallel corpora being rather short. It is unclear whether models trained on only short inputs can perform well on longer inputs.</p>

<div class="no-row-height column-margin column-container"><div class="">
<p>The limited maximum sequence length of multilingual models is discussed in a <a href="https://github.com/UKPLab/sentence-transformers/issues/1476">Github issue</a>.</p>
</div></div><p>In this article, we set out to train a new model where we investigate and try to address the sequence length problem.</p>
</section>
<section id="longer-parallel-texts" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="longer-parallel-texts">Longer parallel texts</h2>
<p>The new model is trained on a similar set of source datasets as the old one, assembled mostly from the <a href="https://opus.nlpl.eu/">Open Parallel Corpus (OPUS)</a>. We use <code>JW300</code>, <code>Europarl</code>, <code>DGT-TM</code>, <code>EMEA</code>, <code>ELITR-ECA</code>, <code>TED2020</code>, <code>Tatoeba</code> and <code>OpenSubtitles</code>. We discovered issues in alignments in the published files from OPUS for the DGT dataset. For this reason we downloaded all the raw DGT-TM data files from EU’s data portal and processed them ourselves (<a href="https://github.com/kb-labb/swedish-sbert/blob/main/get_parallel_data_dgt.py">code</a>). Some of the datasets have quality filters indicating the confidence of the alignments. You can see which thresholds we used in the following <a href="https://github.com/kb-labb/swedish-sbert/blob/fb829a2ce3832a24912f29b1dc20c7f4878bc44e/get_parallel_data_opus.py#L57-L67">highlighted lines</a>.</p>

<div class="no-row-height column-margin column-container"><div class="">
<p>Read more about the datasets in <a href="https://kb-labb.github.io/posts/2021-08-23-a-swedish-sentence-transformer/#data-parallel-corpus-training-data">the original article</a>.</p>
</div></div><p>The main difference when it comes to training data comes from</p>
<ol type="1">
<li>identifying the datasets where sentences are ordered consecutively.</li>
<li>concatenating consecutive sentences in both English and Swedish to longer parallel texts.</li>
</ol>
<p>A dataset with longer texts was created using <code>DGT-TM</code>, <code>Europarl</code>, <code>EMEA</code>, <code>ELITR-ECA</code>, <code>JW300</code> and <code>TED2020</code>. Word counts were calculated for all sentences. Consecutive sentences were then concatenated until the cumulative number of words exceeded a maximum word limit determined by sampling from a truncated normal distribution:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cmathcal%7BTN%7D(%5Cmu,%20%5Csigma,%20a,%20b)%0A"></p>

<div class="no-row-height column-margin column-container"><div class="">
<p><img src="https://latex.codecogs.com/png.latex?%5Cmu"> – mean.<br>
<img src="https://latex.codecogs.com/png.latex?%5Csigma"> – standard deviation.<br>
<img src="https://latex.codecogs.com/png.latex?a"> – minimum nr of words.<br>
<img src="https://latex.codecogs.com/png.latex?b"> – maximum nr of words.</p>
</div></div><p>where <img src="https://latex.codecogs.com/png.latex?%5Cmu=270">, <img src="https://latex.codecogs.com/png.latex?%5Csigma=110">, <img src="https://latex.codecogs.com/png.latex?a=60">, <img src="https://latex.codecogs.com/png.latex?b=330">. Generally each word is represented by an average of <img src="https://latex.codecogs.com/png.latex?1.3"> to <img src="https://latex.codecogs.com/png.latex?1.4"> tokens. Some of the concatenated texts will thus exceed the <img src="https://latex.codecogs.com/png.latex?384"> max sequence length of the model and be truncated. We imagine truncation will be common in real world usage as well, and the pretraining schemes of some language models such as BERT even included too long sequences for this very reason. Therefore we don’t regard the occasional truncation as an issue.</p>
</section>
<section id="new-models" class="level2">
<h2 class="anchored" data-anchor-id="new-models">New models</h2>
<p>We train two new models with <img src="https://latex.codecogs.com/png.latex?384"> max sequence length. The one called <strong>v1.1</strong> is trained with the same teacher model as our original <strong>v1.0</strong> model. The new default model called <strong>v2.0</strong> is trained with a more recently released teacher model <code>all-mpnet-base-v2</code> that is <a href="https://www.sbert.net/docs/pretrained_models.html">supposedly better</a>. An overview of available models and their differences are listed below.</p>
<table class="caption-top table">
<thead>
<tr class="header">
<th>Model version</th>
<th style="text-align: left;">Teacher Model</th>
<th>Max Sequence Length</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>v1.0</td>
<td style="text-align: left;"><a href="https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2">paraphrase-mpnet-base-v2</a></td>
<td>256</td>
</tr>
<tr class="even">
<td>v1.1</td>
<td style="text-align: left;"><a href="https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2">paraphrase-mpnet-base-v2</a></td>
<td>384</td>
</tr>
<tr class="odd">
<td>v2.0</td>
<td style="text-align: left;"><a href="https://huggingface.co/sentence-transformers/all-mpnet-base-v2">all-mpnet-base-v2</a></td>
<td>384</td>
</tr>
</tbody>
</table>
<style>
table th:first-of-type {
    width: 32%;
}
table th:nth-of-type(2) {
    width: 36%;
}
table th:nth-of-type(3) {
    width: 32%;
}
</style>
<p>You can still access and use older version of the model with Huggingface via</p>
<div id="4bb61f8d" class="cell" data-execution_count="1">
<div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb1-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> transformers <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> AutoModel</span>
<span id="cb1-2">AutoModel.from_pretrained(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'KBLab/sentence-bert-swedish-cased'</span>, revision<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"v1.0"</span>)</span></code></pre></div>
</div>
<p>or with <code>sentence-transformers</code> by cloning the repository to your computer with</p>
<div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb2-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">git</span> clone <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--depth</span> 1 <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--branch</span> v1.0 https://huggingface.co/KBLab/sentence-bert-swedish-cased</span></code></pre></div>
<p>and then pointing to that local folder when loading the model in:</p>
<div id="1c183c59" class="cell" data-execution_count="2">
<div class="sourceCode cell-code" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb3-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> sentence_transformers <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> SentenceTransformer</span>
<span id="cb3-2">model <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> SentenceTransformer(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"path_to_model_folder/sentence-bert-swedish-cased"</span>)</span></code></pre></div>
</div>
</section>
<section id="training" class="level2">
<h2 class="anchored" data-anchor-id="training">Training</h2>
<p>The published models <strong>v1.1</strong>, and <strong>v2.0</strong> were first trained on datasets without concatenations for about 380k steps, since this was 4 times faster than training with longer texts mixed in. After 48 hours (380k steps), they were trained for another ~100k steps (48 hours) with longer paragraphs mixed in.</p>
<p>Initially, training and convergence was very slow when using <code>all-mpnet-base-v2</code> as a teacher model. Our student model had difficulties and took much longer to converge to the same evaluation results compared to when it was trained with <code>paraphrase-mpnet-base-v2</code>. After some troubleshooting involving ablations such as:</p>
<ul>
<li>Training only with parallel sentences.</li>
<li>Training with only longer paragraphs.</li>
<li>Training with a mix.</li>
<li>Repeating training with all the above configurations using <code>paraphrase-mpnet-base-v2</code> as a comparison.</li>
</ul>
<p>eventually we turned to carefully comparing both of the above teacher models for differences. We discovered that <code>all-mpnet-base-v2</code> has an <a href="https://huggingface.co/sentence-transformers/all-mpnet-base-v2/blob/main/modules.json">L2-normalization layer at the end</a> after pooling the embeddings. This normalization layer doesn’t involve any model parameters, it simply rescales the output vector based on the magnitude of its own values. We suspected this layer was making it harder for our student model to learn to output similar embeddings to the teacher model, as the student model now had to learn how to normalize a vector (a process that doesn’t involve model paramters) in addition to emulating the teacher model.</p>
<p>We removed this normalization layer and replaced it with an <code>Identity()</code>-layer (returns the input without any manipulation). After this, the model converged much faster and the evaluation results improved.</p>
</section>
<section id="results-on-superlim" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="results-on-superlim">Results on SuperLim</h2>
<p>To evaluate whether whether training with longer sequences improves model performance, we evaluated the model on two datasets using <a href="https://spraakbanken.gu.se/resurser/superlim">SuperLim</a>, a set of evaluation datasets for Swedish language models. An updated version <strong>v2.0</strong> of SuperLim is in the works, and will be released publicly once it is ready. We luckily had access to a development version of <strong>v2.0</strong>, and evaluated our models on both <strong>v1.0</strong> and <strong>v2.0</strong> of SuperLim.</p>
<section id="sweparahrase-v1.0" class="level3">
<h3 class="anchored" data-anchor-id="sweparahrase-v1.0">SweParahrase v1.0</h3>
<p>The models were evaluated on SweParahrase and SweFAQ. Results from <strong>SweParaphrase v1.0</strong> are displayed below.</p>
<table class="caption-top table">
<thead>
<tr class="header">
<th>Model version</th>
<th>Pearson</th>
<th>Spearman</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>v1.0</td>
<td>0.9183</td>
<td>0.9114</td>
</tr>
<tr class="even">
<td>v1.1</td>
<td>0.9183</td>
<td>0.9114</td>
</tr>
<tr class="odd">
<td>v2.0</td>
<td><strong>0.9283</strong></td>
<td><strong>0.9130</strong></td>
</tr>
</tbody>
</table>
<p><strong>v2.0</strong> of our model inches out a slight win when evaluated on <strong>SweParaphrase v1.0</strong>. However, it should be noted the test set is quite small in this version of SuperLim.</p>
</section>
<section id="sweparaphrase-v2.0" class="level3">
<h3 class="anchored" data-anchor-id="sweparaphrase-v2.0">SweParaphrase v2.0</h3>
<p>Below, we present zero-shot evaluation results on all data splits. They display the model’s performance out of the box, without any fine-tuning.</p>
<table class="caption-top table">
<thead>
<tr class="header">
<th>Model version</th>
<th>Data split</th>
<th>Pearson</th>
<th>Spearman</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>v1.0</td>
<td>train</td>
<td>0.8355</td>
<td>0.8256</td>
</tr>
<tr class="even">
<td>v1.1</td>
<td>train</td>
<td><strong>0.8383</strong></td>
<td><strong>0.8302</strong></td>
</tr>
<tr class="odd">
<td>v2.0</td>
<td>train</td>
<td>0.8209</td>
<td>0.8059</td>
</tr>
<tr class="even">
<td>v1.0</td>
<td>dev</td>
<td>0.8682</td>
<td>0.8774</td>
</tr>
<tr class="odd">
<td>v1.1</td>
<td>dev</td>
<td><strong>0.8739</strong></td>
<td><strong>0.8833</strong></td>
</tr>
<tr class="even">
<td>v2.0</td>
<td>dev</td>
<td>0.8638</td>
<td>0.8668</td>
</tr>
<tr class="odd">
<td>v1.0</td>
<td>test</td>
<td>0.8356</td>
<td>0.8476</td>
</tr>
<tr class="even">
<td>v1.1</td>
<td>test</td>
<td><strong>0.8393</strong></td>
<td><strong>0.8550</strong></td>
</tr>
<tr class="odd">
<td>v2.0</td>
<td>test</td>
<td>0.8232</td>
<td>0.8213</td>
</tr>
</tbody>
</table>
<p>In general, <strong>v1.1</strong>, the model trained using the same teacher model as the original model, but using longer texts, correlates the most with human assessment of text similarity on SweParaphrase v2.0.</p>
</section>
<section id="swefaq-v2.0" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="swefaq-v2.0">SweFAQ v2.0</h3>
<p>When it comes to retrieval tasks, <strong>v2.0</strong> performs the best by quite a substantial margin. It is better at matching the correct answer to a question compared to v1.1 and v1.0. Notably <strong>v1.1</strong> performs better than <strong>v1.0</strong>, with the only difference between the two models being that longer parallel texts were included in the training of <strong>v1.1</strong>.</p>
<table class="caption-top table">
<thead>
<tr class="header">
<th>Model version</th>
<th>Data split</th>
<th>Accuracy</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>v1.0</td>
<td>train</td>
<td>0.5262</td>
</tr>
<tr class="even">
<td>v1.1</td>
<td>train</td>
<td>0.6236</td>
</tr>
<tr class="odd">
<td>v2.0</td>
<td>train</td>
<td><strong>0.7106</strong></td>
</tr>
<tr class="even">
<td>v1.0</td>
<td>dev</td>
<td>0.4636</td>
</tr>
<tr class="odd">
<td>v1.1</td>
<td>dev</td>
<td>0.5818</td>
</tr>
<tr class="even">
<td>v2.0</td>
<td>dev</td>
<td><strong>0.6727</strong></td>
</tr>
<tr class="odd">
<td>v1.0</td>
<td>test</td>
<td>0.4495</td>
</tr>
<tr class="even">
<td>v1.1</td>
<td>test</td>
<td>0.5229</td>
</tr>
<tr class="odd">
<td>v2.0</td>
<td>test</td>
<td><strong>0.5871</strong></td>
</tr>
</tbody>
</table>
<aside>
Examples how to evaluate the newer model on some of the test sets of SuperLim v2.0 can be found on the following links: <a href="https://github.com/kb-labb/swedish-sbert/blob/main/evaluate_faq.py">evaluate_faq.py</a> (Swedish FAQ), <a href="https://github.com/kb-labb/swedish-sbert/blob/main/evaluate_sweparaphrase.py">evaluate_sweparaphrase.py</a>
</aside>
</section>
</section>
<section id="acknowledgements" class="level2">
<h2 class="anchored" data-anchor-id="acknowledgements">Acknowledgements</h2>
<p>We gratefully acknowledge the HPC RIVR consortium (<a href="www.hpc-rivr.si">www.hpc-rivr.si</a>) and EuroHPC JU (<a href="eurohpc-ju.europa.eu">eurohpc-ju.europa.eu</a>) for funding this research by providing computing resources of the HPC system Vega at the Institute of Information Science (<a href="www.izum.si">www.izum.si</a>).</p>
</section>


<div id="quarto-appendix" class="default"><section id="code-availability" class="level2 appendix"><h2 class="anchored quarto-appendix-heading">Code availability</h2><div class="quarto-appendix-contents">

<p>The code used to train and evaluate KBLab’s Swedish Sentence BERT is available at https://github.com/kb-labb/swedish-sbert.</p>


</div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{rekathati2023,
  author = {Rekathati, Faton},
  title = {Swedish {Sentence} {Transformer} 2.0},
  date = {2023-01-16},
  url = {https://kb-labb.github.io/posts/2023-01-16-sentence-transformer-20/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-rekathati2023" class="csl-entry quarto-appendix-citeas">
Rekathati, Faton. 2023. <span>“Swedish Sentence Transformer 2.0.”</span>
January 16, 2023. <a href="https://kb-labb.github.io/posts/2023-01-16-sentence-transformer-20/">https://kb-labb.github.io/posts/2023-01-16-sentence-transformer-20/</a>.
</div></div></section></div> ]]></description>
  <guid>https://kb-labb.github.io/posts/2023-01-16-sentence-transformer-20/</guid>
  <pubDate>Sun, 15 Jan 2023 23:00:00 GMT</pubDate>
  <media:content url="https://kb-labb.github.io/posts/2023-01-16-sentence-transformer-20/images/sentence_transformer.svg" medium="image" type="image/svg+xml"/>
</item>
<item>
  <title>BERTopic for Swedish: Topic modeling made easier via KB-BERT</title>
  <dc:creator>Elena Fano</dc:creator>
  <dc:creator>Chris Haffenden</dc:creator>
  <link>https://kb-labb.github.io/posts/2022-06-14-bertopic/</link>
  <description><![CDATA[ 





<p>The emergence of transformer-based language models has significantly enhanced the potential for machine learning to be used for extracting meaning from large volumes of text. Yet this potential has hitherto been limited to the select few with the requisite knowledge in data science. To the vast majority without prior experience of programming in Python, the promise of cutting-edge language processing from a BERT model remains just that—a distant promise.</p>
<p>However, this has now changed with the development of BERTopic. Recent work by the Dutch data scientist, Maarten Grootendorst makes it possible to harness the impressive performance of BERT language models without first needing to become an expert in data science. More specifically, it enables the sophisticated and flexible NLP capacities of a BERT model to serve as the basis for a more accessible form of topic modeling. By offering a new and simpler way of using KBLab’s language models, BERTopic brings a cutting-edge yet previously technically challenging method within reach of a broader range of researchers and other users working with Swedish material.</p>
<p>In this post, we provide a brief introduction to using BERTopic for topic modeling with Swedish text data. We explain how BERTopic harnesses KBLab’s models to produce state-of-the-art topic models, and we outline some tips on how to get started. If you’re interested in finding out more about this particular method, or about topic modeling in general, check out the links at the bottom of the page!</p>
<div id="fig-intertopic" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-intertopic-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="Bertopic1.png" class="lightbox" data-gallery="quarto-lightbox-gallery-1" title="Figure&nbsp;1: Intertopic distance map for a topic model of Swedish governmental reports (SOUs) on education between 1970-2021"><img src="https://kb-labb.github.io/posts/2022-06-14-bertopic/Bertopic1.png" class="img-fluid figure-img"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-intertopic-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;1: Intertopic distance map for a topic model of Swedish governmental reports (SOUs) on education between 1970-2021
</figcaption>
</figure>
</div>
<section id="what-is-topic-modeling-and-why-might-it-be-helpful-for-researchers" class="level3">
<h3 class="anchored" data-anchor-id="what-is-topic-modeling-and-why-might-it-be-helpful-for-researchers">What is topic modeling and why might it be helpful for researchers?</h3>
<p>Topic modeling is a quantitative method for text analysis that allows researchers to gain insights on large text corpora without manually going through all the data themselves. The principal idea is to find a number of major topics in the collection and to be able to retrieve documents that are relevant for these topics.</p>
<p>The output of a topic model is a list of topics represented by words that define them. It is possible to visualize the topics in charts to see their size and distance to each other (see Figure&nbsp;1).<br>
Topic models can be used in many ways, for example to follow the development of a concept over time or to analyze subtopics of a given issue (see Figure&nbsp;2 and Figure&nbsp;3). The main insight is in utilizing data mining algorithms to make vast amounts of data searchable—and thus legible—according to the particular themes of which it is comprised.</p>
<div id="fig-topicstime" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-topicstime-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="Bertopic4.png" class="lightbox" data-gallery="quarto-lightbox-gallery-2" title="Figure&nbsp;2: Topics over time for model of Swedish governmental reports (SOUs) on education between 1970-2021"><img src="https://kb-labb.github.io/posts/2022-06-14-bertopic/Bertopic4.png" class="img-fluid figure-img"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-topicstime-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;2: Topics over time for model of Swedish governmental reports (SOUs) on education between 1970-2021
</figcaption>
</figure>
</div>
</section>
<section id="whats-new-with-bertopic-and-how-does-it-make-the-method-more-accessible" class="level3">
<h3 class="anchored" data-anchor-id="whats-new-with-bertopic-and-how-does-it-make-the-method-more-accessible">What’s new with BERTopic and how does it make the method more accessible?</h3>
<p>BERTopic is a new topic modeling tool that builds on powerful neural networks. At the base there is a model called sentence-BERT, which can be considered an all-purpose language model that has some understanding of the meaning of a sentence or paragraph. The documents are grouped into clusters and then some words are extracted from each cluster to label the topic.</p>
<p>The main advantage of BERTopic over traditional topic models is that there is next to no pre-processing of the documents required prior to modeling: the pre-trained transformer model, sentence-BERT, takes care of identifying the meaningful parts of the text, which means that we don’t need to do it by hand. On top of that, the process is highly automated, so there are relatively few operative decisions that can influence the final result and that need to be accounted for.</p>
</section>
<section id="what-do-you-need-to-do-to-get-started-with-this-method" class="level3">
<h3 class="anchored" data-anchor-id="what-do-you-need-to-do-to-get-started-with-this-method">What do you need to do to get started with this method?</h3>
<p>To build a BERTopic model you need to have a collection of relatively short documents, for example paragraphs or abstracts. While you don’t need to be an expert, you do need some experience with Python programming, since you need to be able to load your documents into Python to run the modeling on them. If you are planning to use the method on Swedish data, then you may want to use the sentence transformer model we have released, which can be found on KBLab’s HuggingFace page (see link below).</p>
<p>There are excellent tutorials available via the BERTopic repository on Github (link below), which provide the best starting point for testing out the method.</p>
</section>
<section id="what-are-the-computational-requirements-of-this-approach" class="level3">
<h3 class="anchored" data-anchor-id="what-are-the-computational-requirements-of-this-approach">What are the computational requirements of this approach?</h3>
<p>The powerful language model that BERTopic is based on needs to run on a GPU, which not everyone has at their disposal. There are two possible solutions if you do not have access to a GPU at your institution: either you can run BERTopic with less computation-intensive embedding models like Flair, spaCy or Gensim, or you can use Google Colab to have temporary free access to a GPU.</p>
<p>Both options have pros and cons. Using a lighter model that runs on CPU means giving up some of the complex language understanding of a transformer model: BERT and similar models have consistently proven to be superior to other methods in many NLP tasks, and a BERTopic without an underlying BERT would be falling short of its full potential. Using Google Colab, on the other hand, means that you need to have a Google account, you need to have permission to load your documents online and you might not get enough GPU memory and/or time for your modelling needs. But having said that, experimenting with this latter option would certainly be a good place to start!</p>
<div id="fig-hierarchical" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-hierarchical-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="Bertopic2.png" class="lightbox" data-gallery="quarto-lightbox-gallery-3" title="Figure&nbsp;3: Hierarchical clustering of topics for a model of Swedish governmental reports (SOUs) on education between 1970-2021"><img src="https://kb-labb.github.io/posts/2022-06-14-bertopic/Bertopic2.png" class="img-fluid figure-img"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-hierarchical-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;3: Hierarchical clustering of topics for a model of Swedish governmental reports (SOUs) on education between 1970-2021
</figcaption>
</figure>
</div>
</section>
<section id="links-about-bertopic" class="level3">
<h3 class="anchored" data-anchor-id="links-about-bertopic">Links about BERTopic:</h3>
<p>Tutorials on GitHub: https://github.com/MaartenGr/BERTopic<br>
Complete documentation: https://maartengr.github.io/BERTopic/<br>
KBLab’s sentence BERT for Swedish: https://huggingface.co/KBLab/sentence-bert-swedish-cased<br>
Google Colab: https://colab.research.google.com/</p>
</section>
<section id="further-reading-on-topic-modeling-as-a-research-method" class="level3">
<h3 class="anchored" data-anchor-id="further-reading-on-topic-modeling-as-a-research-method">Further reading on topic modeling as a research method:</h3>
<p>“Topic Modeling with BERT,” Medium article by Maarten Grootendorst: https://towardsdatascience.com/topic-modeling-with-bert-779f7db187e6</p>
<p>“Interactive Topic Modeling with BERT,” Medium article by Maarten Grootendorst https://towardsdatascience.com/interactive-topic-modeling-with-bertopic-1ea55e7d73d8</p>
<p>“Probabilistic Topic Models,” David M. Blei article: https://www.cs.columbia.edu/~blei/papers/Blei2012.pdf</p>
<p>“The Digital Humanities Contribution to Topic Modeling”, Journal of Digital Humanities special edition: http://journalofdigitalhumanities.org/2-1/</p>
<p>“Topic Modeling: What Humanists Actually Do With It,” University of California blog post by Teddy Roland: https://digitalhumanities.berkeley.edu/blog/16/07/14/topic-modeling-what-humanists-actually-do-it-guest-post-teddy-roland-university</p>


</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{fano2022,
  author = {Fano, Elena and Haffenden, Chris},
  title = {BERTopic for {Swedish:} {Topic} Modeling Made Easier via
    {KB-BERT}},
  date = {2022-06-14},
  url = {https://kb-labb.github.io/posts/2022-06-14-bertopic/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-fano2022" class="csl-entry quarto-appendix-citeas">
Fano, Elena, and Chris Haffenden. 2022. <span>“BERTopic for Swedish:
Topic Modeling Made Easier via KB-BERT.”</span> June 14, 2022. <a href="https://kb-labb.github.io/posts/2022-06-14-bertopic/">https://kb-labb.github.io/posts/2022-06-14-bertopic/</a>.
</div></div></section></div> ]]></description>
  <guid>https://kb-labb.github.io/posts/2022-06-14-bertopic/</guid>
  <pubDate>Mon, 13 Jun 2022 22:00:00 GMT</pubDate>
  <media:content url="https://kb-labb.github.io/posts/2022-06-14-bertopic/Bertopic_algorithm.png" medium="image" type="image/png" height="127" width="144"/>
</item>
<item>
  <title>Evaluating Swedish Language Models</title>
  <dc:creator>Robin Kurtz</dc:creator>
  <link>https://kb-labb.github.io/posts/2022-03-16-evaluating-swedish-language-models/</link>
  <description><![CDATA[ 





<section id="introduction" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="introduction">Introduction</h2>
<p>In the last couple of years the field of natural language processing (NLP) has seen the rise of larger and larger language models (LMs), such as <a href="https://aclanthology.org/N19-1423.pdf">BERT</a>, <a href="https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf">GPT</a>, and others, based on the <a href="https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf">transformer architecture</a>. These massive models are trained on a simple language modeling task, guessing words using the their previous or surrounding context. Due to the large number of parameters and the amounts of text the models see during training, they learn both syntactic and semantic patterns of words in contexts and thus to some extent language, resembling <a href="https://aclanthology.org/2020.acl-main.463/">natural language understanding (NLU)</a>. From a practical point of view, these models serve as the basis for specialist models that are finetuned to solve a specific task, such as sentiment analysis, named entity recognition, and more. The <a href="https://gluebenchmark.com/">GLUE</a> and <a href="https://super.gluebenchmark.com/">SuperGLUE</a> benchmark suites are a collection of tasks designed to require some form of NLU to outperform a human baseline. With these benchmarks, model creators can easily test the usefulness of their new models on a wide range of tasks, while comparing the performance to other similar models. For the three Scandinavian languages Swedish, Danish, and Norwegian there are so far only few such tasks, that can be used to evaluate a transformer LM’s downstream performance. With the introduction of <em>OverLim</em> we hope to provide a simple benchmark to help assess the quality of the model, while comparing its performance to others. Due to its nature of being an automatically translated dataset, any performance differences should be interpreted carefully, as these models <a href="https://aclanthology.org/2021.nodalida-main.28/">tend to learn</a> with the help of <a href="https://aclanthology.org/N18-2017/">data artifacts</a> unrelated to NLU, of which the automatic translation might introduce even more. A more serious effort of a benchmark suite for Swedish is done in the <a href="https://spraakbanken.gu.se/en/resources/superlim">SuperLim</a> project, which is currently working on creating training data to accompany their test sets.</p>

<div class="no-row-height column-margin column-container"><div class="">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://kb-labb.github.io/posts/2022-03-16-evaluating-swedish-language-models/images/bokstaver_kvalitet.jpg" class="img-fluid figure-img"></p>
<figcaption>Quality! Photographer: Jens Gustavsson/KB</figcaption>
</figure>
</div>
</div></div></section>
<section id="creation-and-content" class="level2">
<h2 class="anchored" data-anchor-id="creation-and-content">Creation and Content</h2>
<p>From the 10 main GLUE tasks and the 8 SuperGLUE tasks (not counting the diagnostic datasets), we choose the following 11 tasks:</p>
<ul>
<li>MNLI – <a href="https://cims.nyu.edu/~sbowman/multinli/">Multi NLI</a></li>
<li>MRPC – <a href="https://aclanthology.org/I05-5002/">Microsoft Reasearch Paraphrase Corpus</a></li>
<li>QNLI – <a href="https://rajpurkar.github.io/SQuAD-explorer/">Question-answering NLI</a></li>
<li>QQP – <a href="https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs">Quora Question Pairs</a></li>
<li>RTE – <a href="https://arxiv.org/pdf/1804.07461.pdf">Recognizing Textual Entailment</a></li>
<li>SST – <a href="https://nlp.stanford.edu/sentiment/treebank.html">Stanford Sentiment Treebank</a></li>
<li>STS-B – <a href="http://ixa2.si.ehu.eus/stswiki/index.php/STSbenchmark">Semantic Textual Similarity Benchmark</a></li>
<li>WNLI – <a href="https://cs.nyu.edu/~davise/papers/WinogradSchemas/WS.html">Winograd NLI</a></li>
<li>BoolQ – <a href="https://arxiv.org/abs/1905.10044">Boolean Questions</a></li>
<li>CB – <a href="https://github.com/mcdm/CommitmentBank">Commitment Bank</a></li>
<li>COPA – <a href="https://people.ict.usc.edu/~gordon/copa.html">Choice of Plausible Alternatives</a></li>
</ul>
<p>The translations were done using <a href="https://marian-nmt.github.io/">Marian-NMT</a> and the models for Swedish, Danish, and Norwegian bokmål provided by <a href="https://github.com/Helsinki-NLP/Opus-MT">Opus-MT</a>.</p>
<p>The original GLUE and SuperGLUE datasets use a test set, which does not have its labels published, as competitors send in their results on the test set, while the benchmark maintainers then evaluate on the hidden labels to avoid cheating by pre-evaluating on the test set, or even training on it. This means that we only have labels for the training and development splits. We use the former development split as the new test split and divide the training split with an 80-20 distribution into a new training and development split. Comparing the results to the English GLUE and SuperGLUE results is therefore difficult, given the smaller training set as well as the different test set.</p>
</section>
<section id="evaluation" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="evaluation">Evaluation</h2>
<p>Together this new dataset we want to give potential users some simple baselines and an easy way to evaluate their own models, and present some of our new and upcoming work-in-progress.</p>
<p>We evaluate the following models:</p>
<ul>
<li><a href="https://huggingface.co/AI-Nordics/bert-large-swedish-cased">AI-Sweden BERT-large</a></li>
<li><a href="https://huggingface.co/bert-base-multilingual-cased">mBERT-base</a></li>
<li><a href="https://huggingface.co/KB/electra-base-swedish-cased-discriminator">ELECTRA-base</a></li>
<li><a href="https://huggingface.co/KBLab/sentence-bert-swedish-cased">Sentence-BERT</a></li>
<li><a href="https://huggingface.co/KBLab/bart-base-swedish-cased">KB-BART</a></li>
<li><a href="https://huggingface.co/KB/bert-base-swedish-cased">KB-BERT</a></li>
<li><a href="https://huggingface.co/KBLab/bert-base-swedish-cased-new">BERT 🤗</a></li>
<li><a href="https://huggingface.co/KBLab/megatron-bert-base-swedish-cased-new">Megatron-BERT-base 125k</a></li>
<li><a href="https://huggingface.co/KBLab/megatron-bert-base-swedish-cased-600k">Megatron-BERT-base 600k</a></li>
<li><a href="https://huggingface.co/KBLab/megatron-bert-large-swedish-cased-110k">Megatron-BERT-large 110k</a></li>
</ul>
<p>KB-BART and the last four models in that list use the same amount of data, roughly 70GB. BERT 🤗 was trained using the huggingface 🤗 framework, concatenating multiple short documents into one long sequence, additionally to the traditional splitting of documents longer than 512 tokens into multiple sequences. The two Megatron-BERT-base models use the same setup, with one being trained for 125k steps and the other for 600k steps, to test the impact of longer training times. Finally, our large BERT model, also using the Megatron-LM framework, is only an intermediate model checkpoint. The intended training time is set to 500k steps and will be continued in the foreseeable future.</p>
<p>Each model was trained and evaluated five times with varying seeds. The final models were chosen according to their performance on the development set. The results below on the test set can therefore in some cases be lower than if the best model had been chosen directly with respect to test-set performance. This does not change the order of the best models with the exception for the RTE task. Some of the sub-tasks return two evaluation measures, accuracy and F-score, which are averaged here for simplicity. For the moment we do not evaluate on COPA.</p>
<p>The training- and evaluation script can be downloaded via the <a href="https://github.com/kb-labb/overlim_eval">git repository</a>.</p>
<div class="column-body-outset">
<table class="caption-top table">
<colgroup>
<col style="width: 18%">
<col style="width: 8%">
<col style="width: 8%">
<col style="width: 8%">
<col style="width: 8%">
<col style="width: 8%">
<col style="width: 8%">
<col style="width: 8%">
<col style="width: 8%">
<col style="width: 8%">
<col style="width: 8%">
</colgroup>
<thead>
<tr class="header">
<th>Model</th>
<th>mnli</th>
<th>mrpc</th>
<th>qnli</th>
<th>qqp</th>
<th>rte</th>
<th>sst</th>
<th>stsb</th>
<th>wnli</th>
<th>boolq</th>
<th>cb</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>AI-Sweden BERT-large</td>
<td>83.49%</td>
<td><strong>86.67%</strong></td>
<td>90.10%</td>
<td>68.69%</td>
<td><strong>70.40%</strong></td>
<td>91.28%</td>
<td>87.61%</td>
<td>46.48%</td>
<td>70.28%</td>
<td>59.14%</td>
</tr>
<tr class="even">
<td>mBERT-base</td>
<td>78.40%</td>
<td>81.73%</td>
<td>86.97%</td>
<td>69.17%</td>
<td>67.15%</td>
<td>89.22%</td>
<td>81.93%</td>
<td>43.66%</td>
<td>65.96%</td>
<td>58.95%</td>
</tr>
<tr class="odd">
<td>ELECTRA-base</td>
<td>78.18%</td>
<td>76.66%</td>
<td>84.07%</td>
<td>60.68%</td>
<td>54.51%</td>
<td>88.19%</td>
<td>10.54%</td>
<td><strong>60.56%</strong></td>
<td>62.35%</td>
<td>56.08%</td>
</tr>
<tr class="even">
<td>Sentence-BERT</td>
<td>81.60%</td>
<td>76.22%</td>
<td>86.73%</td>
<td>69.67%</td>
<td>51.26%</td>
<td>90.25%</td>
<td>82.81%</td>
<td>42.25%</td>
<td>67.43%</td>
<td>59.16%</td>
</tr>
<tr class="odd">
<td>KB-BART</td>
<td>79.60%</td>
<td>73.41%</td>
<td>85.47%</td>
<td></td>
<td>53.07%</td>
<td>89.33%</td>
<td>74.33%</td>
<td>45.07%</td>
<td>63.33%</td>
<td>55.81%</td>
</tr>
<tr class="even">
<td>KB-BERT</td>
<td>80.97%</td>
<td>83.53%</td>
<td>89.27%</td>
<td>70.21%</td>
<td>65.34%</td>
<td>90.83%</td>
<td>87.42%</td>
<td>38.03%</td>
<td>67.31%</td>
<td>57.66%</td>
</tr>
<tr class="odd">
<td>BERT 🤗</td>
<td>81.15%</td>
<td>76.75%</td>
<td>87.63%</td>
<td>62.65%</td>
<td>52.35%</td>
<td>90.71%</td>
<td>54.83%</td>
<td>47.89%</td>
<td>64.98%</td>
<td><strong>60.44%</strong></td>
</tr>
<tr class="even">
<td>Megatron-BERT-base 125k</td>
<td>80.23%</td>
<td>78.40%</td>
<td>88.38%</td>
<td>73.73%</td>
<td>65.34%</td>
<td>88.88%</td>
<td>83.61%</td>
<td>50.70%</td>
<td>64.98%</td>
<td>59.19%</td>
</tr>
<tr class="odd">
<td>Megatron-BERT-base 600k</td>
<td>82.48%</td>
<td>76.34%</td>
<td>89.13%</td>
<td><strong>75.56%</strong></td>
<td>63.90%</td>
<td>90.37%</td>
<td>77.46%</td>
<td>40.85%</td>
<td>62.39%</td>
<td>57.61%</td>
</tr>
<tr class="even">
<td>KB BERT-large 110k</td>
<td><strong>84.50%</strong></td>
<td>81.36%</td>
<td><strong>91.12%</strong></td>
<td>72.44%</td>
<td>69.31%</td>
<td><strong>93.00%</strong></td>
<td><strong>87.75%</strong></td>
<td>29.58%</td>
<td><strong>72.23%</strong></td>
<td>54.63%</td>
</tr>
</tbody>
</table>
</div>
<p>The two large models mostly come out on top, with some exceptions achieved by the new BERT-base models. All models fail on the WNLI set, with the exception of ELECTRA-base, which in turn underperforms on everything else; performance on this sub-task should therefore be taken (even more than the others) with a grain of salt. The MRPC and RTE tasks show that our models do not perform, compared to the large AI-Sweden BERT and Google’s multilingual BERT, well there. We believe that this might be due to difference in training data used, which relies much less on data crawled from the web.</p>
<section id="double-check-performance-on-other-datasets" class="level3">
<h3 class="anchored" data-anchor-id="double-check-performance-on-other-datasets">Double-Check Performance on other Datasets</h3>
<p>When evaluating models it is very important to test them on a wide variety of tasks. While it is tempting to test models on a test-suite like (Super)GLUE or OverLim, these in particular focus on one type of application. Large LMs also benefit other applications considered more basic such as tagging or parsing. Users should not choose their model because of some over-optimized aggregate of task-irrelevant scores, that favors larger models, which might even be too unwieldy for production purposes. When evaluating some of the models on our <a href="https://kb-labb.github.io/posts/2022-02-07-sucx3_ner/">SUCX 3.0 - NER</a> dataset, we see this clearly. For these experiments we again trained and evaluated each model five times, on the mixed-case variations of the <em>SUCX 3.0 - NER</em> dataset using both the <em>original</em> tags as well as the more stable <em>simple</em> ones. The reported F-scores on the test set were averaged to give a more even view.</p>
<table class="caption-top table">
<thead>
<tr class="header">
<th>Model</th>
<th>Simple</th>
<th>Original</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>AI-Sweden BERT-large</td>
<td>88.73%</td>
<td>86.07%</td>
</tr>
<tr class="even">
<td>KB-BERT</td>
<td>89.35%</td>
<td>86.71%</td>
</tr>
<tr class="odd">
<td>BERT 🤗</td>
<td><strong>89.80%</strong></td>
<td><strong>87.43%</strong></td>
</tr>
<tr class="even">
<td>Megatron-BERT-base 125k</td>
<td>87.84%</td>
<td>84.95%</td>
</tr>
<tr class="odd">
<td>Megatron-BERT-base 600k</td>
<td>88.60%</td>
<td>86.11%</td>
</tr>
<tr class="even">
<td>Megatron-BERT-large 110k</td>
<td>89.78%</td>
<td>87.32%</td>
</tr>
</tbody>
</table>
<p>Here we can see that additional training time increases the performance for the two Megatron-BERT-base models, but even though they have been trained for longer and on more data, the small model differences let them stay behind the <em>old</em> KB-BERT. AI-Sweden’s Megatron-BERT-large model also performs worse than the KB-BERT, while our Megatron-BERT-large manages to barely outperform its smaller predecessor. The winner in this little competition however is the new BERT 🤗 which was also only trained for 125k steps, showing that there is no one best model.</p>
<p>One important aspect when comparing models has been omitted so far. Statistical significance testing is an important tool that is very much underused (such as here). With significance testing we can get a much better idea on which performance differences actually matter, or whether some model only performs consistently better, by already having seen five test-set examples during pretraining.</p>
<p>We invite everyone to test their models, optimize performance of already evaluated models, and compare, even across languages to better understand the strengths and weaknesses of language models.</p>
</section>
</section>
<section id="conclusion" class="level2">
<h2 class="anchored" data-anchor-id="conclusion">Conclusion</h2>
<p>With the publication of this dataset we hope to give the Scandinavian NLP community a new tool for evaluating their language models. Even though we think that any results on this data should not be used to claim the state-of-the-art of your newest model, we hope to instill a little bit of healthy competition into the community, especially for everyone developing multi-lingual models.</p>
</section>
<section id="acknowledgements" class="level2">
<h2 class="anchored" data-anchor-id="acknowledgements">Acknowledgements</h2>
<p>We gratefully acknowledge the HPC RIVR consortium (https://www.hpc-rivr.si) and EuroHPC JU (https://eurohpc-ju.europa.eu) for funding this research by providing computing resources of the HPC system Vega at the Institute of Information Science (https://www.izum.si).</p>


</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{kurtz2022,
  author = {Kurtz, Robin},
  title = {Evaluating {Swedish} {Language} {Models}},
  date = {2022-03-16},
  url = {https://kb-labb.github.io/posts/2022-03-16-evaluating-swedish-language-models/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-kurtz2022" class="csl-entry quarto-appendix-citeas">
Kurtz, Robin. 2022. <span>“Evaluating Swedish Language Models.”</span>
March 16, 2022. <a href="https://kb-labb.github.io/posts/2022-03-16-evaluating-swedish-language-models/">https://kb-labb.github.io/posts/2022-03-16-evaluating-swedish-language-models/</a>.
</div></div></section></div> ]]></description>
  <guid>https://kb-labb.github.io/posts/2022-03-16-evaluating-swedish-language-models/</guid>
  <pubDate>Tue, 15 Mar 2022 23:00:00 GMT</pubDate>
  <media:content url="https://kb-labb.github.io/posts/2022-03-16-evaluating-swedish-language-models/images/bokstaver_kvalitet.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Swedish Bootleg model</title>
  <dc:creator>Elena Fano</dc:creator>
  <link>https://kb-labb.github.io/posts/2022-03-03-swedish-bootleg-model/</link>
  <description><![CDATA[ 





<section id="what-is-bootleg" class="level2">
<h2 class="anchored" data-anchor-id="what-is-bootleg">What is Bootleg?</h2>
<p>Named Entity Disambiguation (NED) is the task to determine which specific entity in a knowledge base a detected named entity refers to. For example, take the sentence “I bought a new Lincoln”: given Wikipedia as a knowledge base, the task of the NED system is to link the named entity “Lincoln” to the page “Lincoln Motor Company” and not to “Lincoln, Nebraska” or “Abraham Lincoln”. This is a rather complex task, because there is very little context to base the disambiguation on, and nowhere in the sentence there is any reference to cars. It is especially tricky to disambiguate rare entities, i.e.&nbsp;entities that appear very few times in the data or not at all.</p>
<p>Yet there is very little doubt in any human mind that I didn’t buy an American president or a city in Nebraska. So how do humans do it? We base our disambiguation on the fact that you don’t buy people or cities, you buy objects. In more abstract terms, we base our decision on the type of the entity, and other similar reasoning patterns.</p>
<p>Bootleg is NED system that aims to improve disambiguation on rare entities, so-called “tail” entities (because they constitute the long tail of the frequency distribution). Developed by researchers at Hazy Research Lab, Bootleg is a transformer-based system that introduces reasoning patterns based on entity type and relations in the disambiguation process. Even if an entity is rare, its type probably isn’t; for example names of different car brands tend to be used much the same way in natural language. Similarly, some adjectives only refer to specific types of entities: you wouldn’t say that a person is long, but you can say that they are tall.</p>
<div id="fig-bootleg-architecture" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-bootleg-architecture-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="bootleg_architecture.png" class="lightbox" data-gallery="quarto-lightbox-gallery-1" title="Figure&nbsp;1: Bootleg architecture"><img src="https://kb-labb.github.io/posts/2022-03-03-swedish-bootleg-model/bootleg_architecture.png" class="img-fluid figure-img"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-bootleg-architecture-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;1: Bootleg architecture
</figcaption>
</figure>
</div>
<p>Bootleg is trained on Wikidata and Wikipedia, that exist for a variety of the world’s languages, and achieves much better results on tail entities compared to a vanilla BERT-based entity disambiguation system (see Figure&nbsp;1). It is an end-to-end system that can detect named entities in running text and disambiguate them, giving as output the unique identifiers of the candidate entities in the knowledge base.</p>
<p>The original Bootleg paper can be found <a href="https://arxiv.org/pdf/2010.10363.pdf">here</a>, as well the official <a href="https://github.com/HazyResearch/bootleg">repository</a> and the <a href="http://ai.stanford.edu/blog/bootleg/">blog post</a> from the Stanford AI Lab.</p>
</section>
<section id="training-and-evaluating-the-swedish-model" class="level2">
<h2 class="anchored" data-anchor-id="training-and-evaluating-the-swedish-model">Training and evaluating the Swedish model</h2>
<p>Since the only pre-trained Bootleg model available from Hazy Research is an English model, we decided to train a Swedish Bootleg on the Swedish part of Wikipedia. The authors of the paper provided us with data pre-processing and training scripts, and the system is also well documented <a href="https://bootleg.readthedocs.io/en/latest/index.html">here</a>. Since Swedish Wikipedia is much smaller than the English (1.5 GB versus 20 GB compressed size), we were able to train the Swedish Bootleg on a single GPU in about two to three days, but we also expected to get a worse model.</p>
<p>For evaluation we also used the scripts provided by the authors and a test set consisting of Wikipedia articles that the model had never seen before. We obtained the following results:</p>
<p>precision = 2024911 / 2052099 = 0.987<br>
recall = 2024911 / 2067105 = 0.979<br>
f1 = 0.983</p>
<p>These results might seem very good but they are somewhat misleading for a couple of reasons. First, this is exactly the same type of material that the model was trained on, so of course the style and register are very similar and this makes the inference task easier. Second, for any entity linking system there is a fair share of entities that only have one possible candidate in the knowledge base, which means that the model will be right every time on these entities regardless of how good it actually is.</p>
<p>A more accurate sense of how well the model performs can be gained by manually testing it on different kinds of material and observing the resulting mistakes. The first step in predicting new data is isolating the mentions to be linked, and this is done through the mention extraction pipeline. This finds entities in the text that correspond to candidate entities in the knowledge base, and ensures that every entity selected for linking has at least one candidate to link to. We substituted the default English spaCy model with our own Swedish model and ran the inference on some test sentences. We observed that the model can indeed perform type disambiguation, as in the following example:</p>
<div id="fig-bootleg-lincoln" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-bootleg-lincoln-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="lincoln.png" class="lightbox" data-gallery="quarto-lightbox-gallery-2" title="Figure&nbsp;2: Bootleg predictions for the sentence “I bought a new Lincoln”"><img src="https://kb-labb.github.io/posts/2022-03-03-swedish-bootleg-model/lincoln.png" class="img-fluid figure-img"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-bootleg-lincoln-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;2: Bootleg predictions for the sentence “I bought a new Lincoln”
</figcaption>
</figure>
</div>
<p>The model understands that the first sentence must be about a person and links to “Abraham Lincoln”, whereas the second is probably about an object and picks the automobile brand “Lincoln”, which is correct. We report some more examples of sentences and the predicted entities to show the model’s behavior and follow up with some comments.</p>
<p><strong>Sentence</strong>: “Duplantis belönades med Svenska Dagbladets bragdguld 2020 för sina dubbla världsrekord och för att dessutom ha noterat det högsta hoppet någonsin utomhus, 6,15.”<br>
<strong>Entities</strong>: [‘Andreas Duplantis’, ‘Svenska Dagbladets guldmedalj’, ‘Friidrottsrekord’, ‘Hopp (dygd)’]</p>
<p><strong>Sentence</strong>: “Ekonomer vid internationella valutafonden IMF har räknat ut att varje storvuxen bardval enbart genom sin förmåga att binda kol utför en nytta värd minst 20 miljoner kronor.”<br>
<strong>Entities</strong>: [‘Internationella valutafonden’, ‘Internationella valutafonden’, ‘Bardvalar’, ‘Kol (bränsle)’, ‘Svensk krona’]</p>
<p><strong>Sentence</strong>: “Elhandelsbolagen i Norden köper sin el på den nordiska elhandelsbörsen Nord Pool och vad det kostar beror på en mängd olika faktorer.”<br>
<strong>Entities</strong>: [‘Norden’, ‘Nord Pool’]</p>
<p>Clearly the main problem is that as long as there is an entry for it in Wikidata, anything is considered an entity. Depending on the application, we might not want everyday objects and concepts like “kol” and “friidrottsrekord” to be treated as name entities. We have already applied a filter that removes from the entity database those aliases that appear very often in Wikipedia texts without being marked as mentions, the reasoning being that these aliases point to common words that are not named entities. Still, the results are a bit noisy. This issue could be probably solved by running a Named Entity Recognition model on the same data, for example kb-bert-ner, and ignoring all the entities in the Bootleg output that the NER model doesn’t recognize as such.</p>
<p>There are also some actual mistakes, for example predicting “Andreas Duplantis” instead of his brother Armand Duplantis, but it would require too much world knowledge for the model to know which one of the two pole-vaulting brothers recently set a world record. In the same sentence, “hopp” is identified as the Christian virtue of “hope”, which is incorrect, but maybe justified by the fact that “högsta hoppet” (highest hope/jump) can be a good collocation for both senses of the word.</p>
</section>
<section id="usage-and-examples" class="level2">
<h2 class="anchored" data-anchor-id="usage-and-examples">Usage and examples</h2>
<p>We are working on making the Swedish model and the entity database available for download, as well as providing ready-made scripts for inference in multilingual settings. Stay tuned for more information!</p>
</section>
<section id="acknowledgements" class="level2">
<h2 class="anchored" data-anchor-id="acknowledgements">Acknowledgements</h2>
<p>Many thanks to <a href="https://cs.stanford.edu/people/lorr1/">Laurel Orr</a> at the <a href="https://hazyresearch.stanford.edu/">Hazy Research Lab</a> for assisting in every step of the data processing and training, both with scripts and insightful tips.</p>


</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{fano2022,
  author = {Fano, Elena},
  title = {Swedish {Bootleg} Model},
  date = {2022-03-03},
  url = {https://kb-labb.github.io/posts/2022-03-03-swedish-bootleg-model/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-fano2022" class="csl-entry quarto-appendix-citeas">
Fano, Elena. 2022. <span>“Swedish Bootleg Model.”</span> March 3, 2022.
<a href="https://kb-labb.github.io/posts/2022-03-03-swedish-bootleg-model/">https://kb-labb.github.io/posts/2022-03-03-swedish-bootleg-model/</a>.
</div></div></section></div> ]]></description>
  <guid>https://kb-labb.github.io/posts/2022-03-03-swedish-bootleg-model/</guid>
  <pubDate>Wed, 02 Mar 2022 23:00:00 GMT</pubDate>
  <media:content url="https://kb-labb.github.io/posts/2022-03-03-swedish-bootleg-model/bootleg_architecture.png" medium="image" type="image/png" height="133" width="144"/>
</item>
<item>
  <title>SUCX 3.0 - NER</title>
  <dc:creator>Robin Kurtz</dc:creator>
  <dc:creator>Joey Öhman</dc:creator>
  <link>https://kb-labb.github.io/posts/2022-02-07-sucx3_ner/</link>
  <description><![CDATA[ 





<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">Introduction</h2>
<div id="fig-ner" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-ner-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<a href="images/ner.png" class="lightbox" data-gallery="quarto-lightbox-gallery-1" title="Figure&nbsp;1: A simple NER example with three entities of two types."><img src="https://kb-labb.github.io/posts/2022-02-07-sucx3_ner/images/ner.png" class="img-fluid figure-img"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-ner-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;1: A simple NER example with three entities of two types.
</figcaption>
</figure>
</div>
<p>Named Entity Recognition (NER), the task of automatically recognizing named entities, such as persons, companies, organizations, etc., is a staple Natural Language Processing (NLP) application. For Swedish, the <a href="https://spraakbanken.gu.se/en/resources/suc3">Stockholm Umeå Corpus (SUC)</a> has been the biggest resource for training NER models, containing more than 30,000 sentences with manually annotated named entities and part-of-speech (POS) tags. Additionally, Språkbanken has further enhanced the corpus with additional syntactic annotations and alternative annotations for NER, both created automatically. This enhanced version of the corpus is called <em>SUCX 3.0</em>. The new named entities largely match the manually annotated ones, use however slightly different categories and introduce two new categories for measurements and time. These new entity annotations are done by a <a href="https://aclanthology.org/L14-1339/">rule-based NER system</a>, creating regular and predictable annotations. A comparison of the entity categories is given below.</p>
<table class="caption-top table">
<thead>
<tr class="header">
<th>Manually Annotated</th>
<th>Automatically Annotated</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>person</td>
<td>PRS</td>
</tr>
<tr class="even">
<td>place</td>
<td>LOC</td>
</tr>
<tr class="odd">
<td>inst</td>
<td>ORG</td>
</tr>
<tr class="even">
<td>work</td>
<td>WRK</td>
</tr>
<tr class="odd">
<td>product</td>
<td>OBJ</td>
</tr>
<tr class="even">
<td>animal</td>
<td>PRS</td>
</tr>
<tr class="odd">
<td>event</td>
<td>EVN</td>
</tr>
<tr class="even">
<td>myth</td>
<td>PRS</td>
</tr>
<tr class="odd">
<td>-</td>
<td>MSR</td>
</tr>
<tr class="even">
<td>-</td>
<td>TME</td>
</tr>
<tr class="odd">
<td>other</td>
<td>-</td>
</tr>
</tbody>
</table>
<p>With large transformer language models (LM) such as <a href="https://arxiv.org/abs/1810.04805">BERT</a> becoming the de-facto standard for most NLP tasks, they have also shown their worth for somewhat simpler tasks such as NER.</p>
<section id="data-splits" class="level3">
<h3 class="anchored" data-anchor-id="data-splits">Data Splits</h3>
<p>In order to compare the performance of various models, it is common in NLP practice to create a canonical split of the training data into training-, development-, and test-data, that everyone uses to train and evaluate their models to provide a fair comparison. While this practice comes with a <a href="https://aclanthology.org/P19-1267/">heap of problems</a>, it is nonetheless an easy way to quickly compare models with the same setup to give an intuition about their performance. We therefore split the SUC 3.0 corpus into these three parts at random, while keeping the distribution of sentences with and without annotations, and the number of named entities per category the same across the three splits.</p>
<p>The original SUC 3.0 corpus uses XML to structure its various types of annotations. We use a much friendlier <code>json</code>-based format that only contains the sentence split into tokens, the POS annotations, and the NER annotations in the BIO format.</p>
<div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode json code-with-copy"><code class="sourceCode json"><span id="cb1-1"><span class="er" style="color: #AD0000;
background-color: null;
font-style: inherit;">e245ac6c-e24b4fe4</span></span>
<span id="cb1-2"><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">[</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"I"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"dag"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"är"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"han"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ingenjör"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"på"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"vetenskapsakademins"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb1-3">                    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"kemisk-tekniska"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"institution"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"i"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Vilnius"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"."</span> <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">]</span></span>
<span id="cb1-4"><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">[</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"PP"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"NN"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"VB"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"PN"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"NN"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"PP"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"NN"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"JJ"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"NN"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"PP"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"PM"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"MAD"</span> <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">]</span></span>
<span id="cb1-5"><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">[</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"B-TME"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"I-TME"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"O"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"O"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"O"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"O"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"O"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"O"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"O"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"O"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"B-LOC"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"O"</span> <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">]</span> </span></code></pre></div>
<p>The data and its variations can be downloaded either with <code>git</code> or directly as a huggingface 🤗 <code>dataset</code> here https://huggingface.co/datasets/KBLab/sucx3_ner.</p>
</section>
<section id="variations" class="level3">
<h3 class="anchored" data-anchor-id="variations">Variations</h3>
<p>The original annotations of SUC are sometimes criticized to be somewhat inconsistent compared with other datasets (should titles be included in the named entity: <em>kungen [Waldemar Atterdag]</em> vs.&nbsp;<em>[kung Carl Gustaf]</em>), to contain needlessly specific categories (animal, myth), and a dangerously confusing (for a machine) <em>other</em> category. In some instances one would therefore prefer the tags automatically annotated by the tagger, over the manual annotations.</p>
<p>In our first dataset variation we only take sentences with annotations, which do not contain the <em>other</em> category, and where the manual and automatic annotations match according to the mapping above. The new <em>measurement</em> (MSR) and <em>time</em> (TME) annotations are included as well. This means that this new dataset is somewhat smaller, as all sentences where the annotations did not match for each token are removed.</p>
<section id="cased-and-uncased" class="level4">
<h4 class="anchored" data-anchor-id="cased-and-uncased">Cased and Uncased</h4>
<p>Due to the custom in Swedish (and a lot of other European languages), to write named entities with a leading capital letter, NER systems quickly learn to rely on this simple feature. This improves performance as it is a clear indicator, when the data consists of properly formatted text, but leads systems to near absolute failure, when the data does not use case, as is often the case in web-text resources or chat.</p>
<p>We therefore also add a completely lower-cased (uncased) version of the dataset, and a cased-uncased-mixed version, that can be used to train and evaluate systems that are supposed to handle more noisy data. The new dataset variations uses the suffix “_lower” at the sentence-id, to indicate that the sentence has been lower-cased. This allows users that wish to train and/or evaluate a model on a dataset where each sentence exists twice, cased and uncased, by simply combining the instances of each dataset into a new one.</p>
</section>
</section>
</section>
<section id="hyper-parameter-optimization" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="hyper-parameter-optimization">Hyper-Parameter Optimization</h2>
<p>While we already achieve good performance with our KB-BERT model finetuned to do NER, we want to see if this can be improved by choosing a different set of hyper-parameters. With Hyper-Parameter Optimization (HPO) methods we can test combinations of different values for a chosen set of hyper-parameters that we believe can impact the performance of our final NER system.</p>
<section id="baselines" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="baselines">Baselines</h3>
<p>First we use the standard set of parameters of the huggingface 🤗 training API to create our baselines. We refer to the manually annotated tags as <code>original</code> (<code>org</code>) and the automatically annotated ones as <code>simple</code>. The measurement used for NER is F1-score, a measure that combines (for each tag) both the precision (to annotate only when it truly is of some type) and recall (to annotate all instances of some type).</p>
<p>Each column illustrates one setting of NER tag &amp; case type with batch size 64 and standard hyperparameters.</p>
<table class="caption-top table">
<colgroup>
<col style="width: 10%">
<col style="width: 14%">
<col style="width: 12%">
<col style="width: 12%">
<col style="width: 18%">
<col style="width: 15%">
<col style="width: 15%">
</colgroup>
<thead>
<tr class="header">
<th></th>
<th>org/cased</th>
<th>org/uncased</th>
<th>org/mixed</th>
<th>simple/cased</th>
<th>simple/uncased</th>
<th>simple/mixed</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>F1-Dev</td>
<td>0.8901</td>
<td>0.8683</td>
<td>0.866</td>
<td>0.9359</td>
<td>0.912</td>
<td>0.9111</td>
</tr>
<tr class="even">
<td>F1-Test</td>
<td>0.8901</td>
<td>0.867</td>
<td>0.8687</td>
<td>0.9346</td>
<td>0.9017</td>
<td>0.9118</td>
</tr>
</tbody>
</table>
<aside>
Note that these experiments use a version for the <em>simple</em> tags that does not use BIO-encoding in combination with the NE-tags. The dataset we publish uses BIO-tags, which can be removed if necessary.
</aside>
<p>While there are many hyperparameters that one can vary, we only vary the learning rate, the weight decay, and the warmup ratio, three closely interconnected training parameters. The hyperparameter space that is shown below, was initially determined through inspiration from published related work and adapted after initial experiments.</p>
<p>The <em>learning rate</em> controls how much parameters are changed at every optimization step. <em>Weight decay</em> denotes a parameter that controls the impact of the regularizing L<sub>2</sub>-norm, favouring models with weights closer to zero. Finally, the <em>warmup ratio</em> controls the length of a warmup period during training in which the learning rate is increased to its initial maximum, aiming to avoid instability during early updates when the model weights are not yet aligned and the learning rate is too large.</p>
<p><strong>Original Hyperparameters:</strong></p>
<pre class="text"><code>Learning Rate: 5e-5
Weight Decay:  0.0
Warmup Ratio:  0.0</code></pre>
<p><strong>Hyperparameter Search Space:</strong></p>
<pre class="text"><code>Learning Rate: [2e-5, 3e-5, ..., 8e-5]
Weight Decay:  [0.00, 0.05, 0.10, 0.15]
Warmup Ratio:  [0.00, 0.04, 0.08, 0.12]</code></pre>
</section>
<section id="methods" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="methods">Methods</h3>
<p>Choosing which hyperparameters to use when training machine learning models often requires a good understanding of the model, the dataset, but also the impact of each of the parameters as well, and how they are connected to each other. Additionally to that, a certain experience is needed as well, and the knowledge of the dark arts of optimization: specific parameter settings that generally work well out of the box, without necessarily being published in some way. For the HPO experiments we used the <a href="https://docs.ray.io/en/latest/tune/index.html"><code>ray tune</code></a> library, which is <a href="https://huggingface.co/blog/ray-tune">easily integrated</a> within the huggingface 🤗 ecosystem.</p>
<p>But even then, there are simply too many possible settings that one can reasonably test manually while making informed decisions based on previous results. One way to solve this is by using grid-search, a method that simply checks every combination of hyperparameters. If the search space is too large, one can instead only search a random sub-space with random search. The following figure illustrates some ways of visualizing the results of hyperparameter optimization, which is useful to gain intuition of the hyperparameter behaviors.</p>
<div class="page-columns page-full">
<div id="fig-hpo-overview" class="quarto-float quarto-figure quarto-figure-center anchored page-columns page-full">
<figure class="quarto-float quarto-float-fig figure page-columns page-full">
<div aria-describedby="fig-hpo-overview-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca" class="page-columns page-full">
<a href="images/RS_simple_lowermix_overall.png" class="lightbox page-columns page-full" data-gallery="quarto-lightbox-gallery-2" title="Figure&nbsp;2: Weights and biases overview over the results from an HPO experiment with the simple lower mix variant of the dataset."><img src="https://kb-labb.github.io/posts/2022-02-07-sucx3_ner/images/RS_simple_lowermix_overall.png" class="img-fluid figure-img column-page"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-hpo-overview-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;2: Weights and biases overview over the results from an HPO experiment with the simple lower mix variant of the dataset.
</figcaption>
</figure>
</div>
</div>
<p>While more advanced methods such as <a href="https://arxiv.org/abs/1810.05934">ASHA</a>, <a href="https://arxiv.org/abs/1807.01774">BOHB</a>, and <a href="https://arxiv.org/abs/1711.09846">PBT</a> are applicable in our setting (and were tested to some degree), it is sufficient to employ random search when choosing a discrete set of values for each hyperparameter, instead of letting the algorithms explore the search space on their own.</p>
<section id="advanced-hpo-methods" class="level4 page-columns page-full">
<h4 class="anchored" data-anchor-id="advanced-hpo-methods">Advanced HPO Methods</h4>
<p>The more advanced HPO algorithms excel in different settings and can provide more efficient optimization. For example, ASHA and BOHB are both scalable and robust and can rapidly search through a large hyperparameter space by heavily utilizing early termination of non-promising runs. Furthermore, Bayesian Optimization (BO) models estimate the objective function and can draw informed samples of hyperparameter configurations, yielding successively better samples throughout the HPO session. While BO and early stopping sound promising, they possess the most value with a large number of runs and when early iterations of training consistently indicate end-of-run performance. In our case, we used a limited number of runs and searched for hyperparameters that clearly do not benefit from early stopping. For instance, a low learning rate with a large warmup ratio might perform poorly after training only for an epoch but may result in good performance after the full run.</p>
<p>The figure below demonstrates the early stopping behavior used in ASHA, terminating the lowest performing runs after roughly 2 epochs. Note that mechanisms as early stopping makes it difficult to visualize results and gain intuition about hyperparameters, as the final F1-scores become strongly biased towards runs that make it through the early phases of training.</p>
<div class="page-columns page-full">
<div id="fig-asha-original-lower" class="quarto-float quarto-figure quarto-figure-center anchored page-columns page-full">
<figure class="quarto-float quarto-float-fig figure page-columns page-full">
<div aria-describedby="fig-asha-original-lower-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca" class="page-columns page-full">
<a href="images/ASHA_original_lower_f1.png" class="lightbox page-columns page-full" data-gallery="quarto-lightbox-gallery-3" title="Figure&nbsp;3: The validation F1 scores for an HPO session using ASHA, searching hyperparameters for the original lower case variation of the dataset."><img src="https://kb-labb.github.io/posts/2022-02-07-sucx3_ner/images/ASHA_original_lower_f1.png" class="img-fluid figure-img column-body-outset"></a>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-asha-original-lower-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;3: The validation F1 scores for an HPO session using ASHA, searching hyperparameters for the original lower case variation of the dataset.
</figcaption>
</figure>
</div>
</div>
<p>DeepMind’s PBT proved highly resource-demanding in our experiments and showed little to no performance gain. Increasing the population size and tweaking with other settings may yield a strong hyperparameter schedule if one has access to compute and wants to push the most out of a model/dataset. We did not investigate this further. Our experiments showed no significant advantage to using these advanced methods over random search.</p>
<p>We also conducted some modest experiments tuning the attention dropout rate, hidden dropout rate, and random seed as well, together with the more advanced HPO methods. Those results indicated that we can squeeze out a small amount of performance through more intricate HPO, but the procedure becomes expensive and prone to overfitting on the validation set without observing any performance gains on the test set.</p>
</section>
</section>
<section id="results" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="results">Results</h3>
<p>We report our results using random search with 30 trials training on both the <em>original</em> and <em>simple</em> tags. For each tag-family we train the taggers on cased, uncased, and a mixed data set, while evaluating on all three development and test sets plus an additional set where only the named entity has been lower-cased. The columns labelled <em>uncased-cased-both</em> and <em>ne-lower-cased-both</em> denote data sets, in which every sentence appears both cased and uncased.</p>
<p>Most notably in these results is the performance of a system trained on regular <em>cased</em> data, when evaluated on <em>uncased</em> or partially lowercased data. At the same time we see that the system trained on a mix of cased and uncased data performs only slightly worse than their pure counterparts on the pure evaluation sets, while clearly outperforming them on the mixed evaluation sets.</p>
<section id="development" class="level4 page-columns page-full">
<h4 class="anchored" data-anchor-id="development">Development</h4>
<div class="column-body-outset">
<table class="caption-top table">
<colgroup>
<col style="width: 7%">
<col style="width: 13%">
<col style="width: 5%">
<col style="width: 3%">
<col style="width: 5%">
<col style="width: 13%">
<col style="width: 14%">
<col style="width: 6%">
<col style="width: 14%">
<col style="width: 15%">
</colgroup>
<thead>
<tr class="header">
<th>Tag Family</th>
<th>Trained on</th>
<th>HPO Alg</th>
<th>cased</th>
<th>uncased</th>
<th>uncased-cased-mix</th>
<th>uncased-cased-both</th>
<th>ne-lower</th>
<th>ne-lower-cased-mix</th>
<th>ne-lower-cased-both</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Original</td>
<td>cased</td>
<td>RS</td>
<td>0.8951</td>
<td>0.4067</td>
<td>0.7054</td>
<td>0.6987</td>
<td>0.4110</td>
<td>0.7045</td>
<td>0.6985</td>
</tr>
<tr class="even">
<td>Original</td>
<td>uncased</td>
<td>RS</td>
<td>0.7847</td>
<td>0.8713</td>
<td>0.8278</td>
<td>0.8293</td>
<td>0.8695</td>
<td>0.8263</td>
<td>0.8285</td>
</tr>
<tr class="odd">
<td>Original</td>
<td>uncased-cased-mix</td>
<td>RS</td>
<td>0.8821</td>
<td>0.8573</td>
<td>0.8702</td>
<td>0.8698</td>
<td>0.8504</td>
<td>0.8671</td>
<td>0.866</td>
</tr>
<tr class="even">
<td>Simple</td>
<td>cased</td>
<td>RS</td>
<td>0.9345</td>
<td>0.3037</td>
<td>0.7035</td>
<td>0.6974</td>
<td>0.3038</td>
<td>0.6995</td>
<td>0.6941</td>
</tr>
<tr class="odd">
<td>Simple</td>
<td>uncased</td>
<td>RS</td>
<td>0.8361</td>
<td>0.9157</td>
<td>0.8753</td>
<td>0.8774</td>
<td>0.9154</td>
<td>0.8754</td>
<td>0.8773</td>
</tr>
<tr class="even">
<td>Simple</td>
<td>uncased-cased-mix</td>
<td>RS</td>
<td>0.9275</td>
<td>0.9078</td>
<td>0.9185</td>
<td>0.9177</td>
<td>0.9029</td>
<td>0.9155</td>
<td>0.9153</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="test" class="level4 page-columns page-full">
<h4 class="anchored" data-anchor-id="test">Test</h4>
<div class="column-body-outset">
<table class="caption-top table">
<colgroup>
<col style="width: 7%">
<col style="width: 13%">
<col style="width: 5%">
<col style="width: 3%">
<col style="width: 5%">
<col style="width: 13%">
<col style="width: 14%">
<col style="width: 6%">
<col style="width: 14%">
<col style="width: 15%">
</colgroup>
<thead>
<tr class="header">
<th>Tag Family</th>
<th>Trained on</th>
<th>HPO Alg</th>
<th>cased</th>
<th>uncased</th>
<th>uncased-cased-mix</th>
<th>uncased-cased-both</th>
<th>ne-lower</th>
<th>ne-lower-cased-mix</th>
<th>ne-lower-cased-both</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Original</td>
<td>cased</td>
<td>RS</td>
<td>0.8978</td>
<td>0.4053</td>
<td>0.6940</td>
<td>0.7000</td>
<td>0.4103</td>
<td>0.6924</td>
<td>0.6998</td>
</tr>
<tr class="even">
<td>Original</td>
<td>uncased</td>
<td>RS</td>
<td>0.7811</td>
<td>0.8656</td>
<td>0.8248</td>
<td>0.8245</td>
<td>0.8649</td>
<td>0.8245</td>
<td>0.8242</td>
</tr>
<tr class="odd">
<td>Original</td>
<td>uncased-cased-mix</td>
<td>RS</td>
<td>0.8833</td>
<td>0.8523</td>
<td>0.8661</td>
<td>0.8680</td>
<td>0.8489</td>
<td>0.8650</td>
<td>0.8663</td>
</tr>
<tr class="even">
<td>Simple</td>
<td>cased</td>
<td>RS</td>
<td>0.9304</td>
<td>0.2963</td>
<td>0.6940</td>
<td>0.6929</td>
<td>0.2902</td>
<td>0.6879</td>
<td>0.6861</td>
</tr>
<tr class="odd">
<td>Simple</td>
<td>uncased</td>
<td>RS</td>
<td>0.8299</td>
<td>0.9075</td>
<td>0.8687</td>
<td>0.8702</td>
<td>0.9074</td>
<td>0.8685</td>
<td>0.8702</td>
</tr>
<tr class="even">
<td>Simple</td>
<td>uncased-cased-mix</td>
<td>RS</td>
<td>0.9219</td>
<td>0.8988</td>
<td>0.9083</td>
<td>0.9104</td>
<td>0.8950</td>
<td>0.9064</td>
<td>0.9085</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="baselines-comparison" class="level4 page-columns page-full">
<h4 class="anchored" data-anchor-id="baselines-comparison">Baselines Comparison</h4>
<p>In order to see how much HPO actually helps for this task we compare with our unoptimized baseline systems. Each column illustrates one setting of tag &amp; case type with the performance difference after HPO: <code>F1(HPO) - F1(baseline)</code></p>
<div class="column-body-outset">
<table class="caption-top table">
<colgroup>
<col style="width: 10%">
<col style="width: 14%">
<col style="width: 12%">
<col style="width: 12%">
<col style="width: 18%">
<col style="width: 15%">
<col style="width: 15%">
</colgroup>
<thead>
<tr class="header">
<th></th>
<th>org/cased</th>
<th>org/uncased</th>
<th>org/mixed</th>
<th>simple/cased</th>
<th>simple/uncased</th>
<th>simple/mixed</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>F1-Dev</td>
<td>+0.005</td>
<td>+0.003</td>
<td>+0.0042</td>
<td>-0.0014</td>
<td>+0.0037</td>
<td>+0.0074</td>
</tr>
<tr class="even">
<td>F1-Test</td>
<td>+0.0077</td>
<td>-0.0014</td>
<td>-0.0026</td>
<td>-0.0042</td>
<td>+0.0058</td>
<td>-0.0035</td>
</tr>
</tbody>
</table>
</div>
<p>Unfortunately these differences are not what we hoped for, pointing to either a bad choice of hyperparameters and values to optimize or the stability of the model being trained regardless of the chosen hyperparameters (up to a certain degree of reasonable values). One such hyperparameter might be the <em>warmup ratio</em>, given that the model is already stable and is only being finetuned to a relatively simple task. We speculate that a more consistent gain in validation set performance could be achieved by taking the stochastic elements into account, i.e.&nbsp;running each experiment with multiple seeds, which argues for the insignificance of the HPO results.</p>
</section>
<section id="successful-hyperparameters" class="level4 page-columns page-full">
<h4 class="anchored" data-anchor-id="successful-hyperparameters">Successful Hyperparameters</h4>
<div class="column-body-outset">
<table class="caption-top table">
<colgroup>
<col style="width: 14%">
<col style="width: 23%">
<col style="width: 9%">
<col style="width: 18%">
<col style="width: 16%">
<col style="width: 16%">
</colgroup>
<thead>
<tr class="header">
<th>Tag Family</th>
<th>Trained on</th>
<th>HPO Alg</th>
<th>learning rate</th>
<th>weight decay</th>
<th>warmup ratio</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Original</td>
<td>cased</td>
<td>RS</td>
<td>7e-05</td>
<td>0.15</td>
<td>0.04</td>
</tr>
<tr class="even">
<td>Original</td>
<td>uncased</td>
<td>RS</td>
<td>5e-05</td>
<td>0.10</td>
<td>0.08</td>
</tr>
<tr class="odd">
<td>Original</td>
<td>uncased-cased-mix</td>
<td>RS</td>
<td>8e-05</td>
<td>0.15</td>
<td>0.12</td>
</tr>
<tr class="even">
<td>Simple</td>
<td>cased</td>
<td>RS</td>
<td>5e-05</td>
<td>0.05</td>
<td>0.04</td>
</tr>
<tr class="odd">
<td>Simple</td>
<td>uncased</td>
<td>RS</td>
<td>8e-05</td>
<td>0.05</td>
<td>0.04</td>
</tr>
<tr class="even">
<td>Simple</td>
<td>uncased-cased-mix</td>
<td>RS</td>
<td>6e-05</td>
<td>0.05</td>
<td>0.12</td>
</tr>
</tbody>
</table>
</div>
</section>
</section>
<section id="models" class="level3">
<h3 class="anchored" data-anchor-id="models">Models</h3>
<p>We publish two models for our <em>simple</em> tags without BIO-encoding, trained on <a href="https://huggingface.co/KBLab/bert-base-swedish-cased-reallysimple-ner">cased data</a> and <a href="https://huggingface.co/KBLab/bert-base-swedish-lowermix-reallysimple-ner">mixed cased-uncased data</a> data for anyone to try, with more models to follow. If you feel that the model underperforms, feel free to continue training it on the validation and test data, or your own personal data. Let us know how the models perform in your projects and how you improved them.</p>
</section>
</section>
<section id="conclusion" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="conclusion">Conclusion</h2>
<p>We have taken the venerable SUC 3.0 dataset and given it a little refresher for people wanting to use its named entity annotations for training and evaluating NER taggers. We hope that the new format and its availability via the huggingface 🤗 ecosystem, together with a suggested train-development-test split will encourage more people to evaluate their models on this task, simply as a downstream finetuning task for large language models or for small specialist models trained to only do NER. With our little excursion to hyperparameter optimization we have learned to use the existing tools to easily find a better fitting set of hyperparameters, while also realizing that the results do not necessarily have to be better than when using the standard set of parameters.</p>
<aside>
The scripts to generate the <em>SUCX 3.0 - NER</em> data from the original data supplied by Språkbanken, as well as the code for HPO and some additional information can be accessed at https://github.com/kb-labb/sucx3_ner.
</aside>
</section>
<section id="acknowledgements" class="level2">
<h2 class="anchored" data-anchor-id="acknowledgements">Acknowledgements</h2>
<p>We gratefully acknowledge the HPC RIVR consortium (https://www.hpc-rivr.si) and EuroHPC JU (https://eurohpc-ju.europa.eu) for funding this research by providing computing resources of the HPC system Vega at the Institute of Information Science (https://www.izum.si).</p>


</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{kurtz2022,
  author = {Kurtz, Robin and Öhman, Joey},
  title = {SUCX 3.0 - {NER}},
  date = {2022-02-07},
  url = {https://kb-labb.github.io/posts/2022-02-07-sucx3_ner/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-kurtz2022" class="csl-entry quarto-appendix-citeas">
Kurtz, Robin, and Joey Öhman. 2022. <span>“SUCX 3.0 - NER.”</span>
February 7, 2022. <a href="https://kb-labb.github.io/posts/2022-02-07-sucx3_ner/">https://kb-labb.github.io/posts/2022-02-07-sucx3_ner/</a>.
</div></div></section></div> ]]></description>
  <guid>https://kb-labb.github.io/posts/2022-02-07-sucx3_ner/</guid>
  <pubDate>Sun, 06 Feb 2022 23:00:00 GMT</pubDate>
  <media:content url="https://kb-labb.github.io/posts/2022-02-07-sucx3_ner/images/ner.png" medium="image" type="image/png" height="26" width="144"/>
</item>
<item>
  <title>KBLab publishes an article about AI in the library in C&amp;RL</title>
  <dc:creator>Elena Fano</dc:creator>
  <dc:creator>Chris Haffenden</dc:creator>
  <link>https://kb-labb.github.io/posts/2022-01-24-kblab-publishes-an-article-in-crl/</link>
  <description><![CDATA[ 





<section id="ai-in-the-library" class="level2">
<h2 class="anchored" data-anchor-id="ai-in-the-library">AI in the library</h2>
<p>With the increasingly rapid appearance of new techniques over the past few years, the role of AI within the library is becoming a hot topic today. Yet despite a growing interest among librarians, there remains surpisingly little research devoted to the subject.<br>
We have sought to fill this gap at KBLab by conducting research focused specifically upon AI in the context of libraries. The aim with our new article, <em>Making and using AI in the library: creating a BERT model at the National Library of Sweden</em>, is to show how AI techniques create new possibilities for processing and organizing information at scale in a library (e.g.&nbsp;though automated classification and enriched metadata), but also how library collections might be used to make even better AI tools in the future.</p>
<p>In this way, we have sought to nuance the image that otherwise tends to predominate of the library as a passive space for the application of AI solutions developed elsewhere. On the contrary, we argue that libraries can have an important roll to play in AI development, especially in relation to language technology (NLP) for other languages than English and Chinese. We hope that our text can lead to new and exciting conversations about the future role of AI in the library!</p>
<section id="read-the-article-online" class="level3">
<h3 class="anchored" data-anchor-id="read-the-article-online">Read the article online</h3>
<p>The article has been accepted for publication in the Open Access journal College &amp; Research Libraries, but is available to read online as a preprint now (<a href="https://osf.io/preprints/socarxiv/k9duq/">link to a downloadable PDF on SocArXiv</a>).</p>


</section>
</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{fano2022,
  author = {Fano, Elena and Haffenden, Chris},
  title = {KBLab Publishes an Article about {AI} in the Library in
    {C\&amp;RL}},
  date = {2022-01-24},
  url = {https://kb-labb.github.io/posts/2022-01-24-kblab-publishes-an-article-in-crl/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-fano2022" class="csl-entry quarto-appendix-citeas">
Fano, Elena, and Chris Haffenden. 2022. <span>“KBLab Publishes an
Article about AI in the Library in C&amp;RL.”</span> January 24, 2022.
<a href="https://kb-labb.github.io/posts/2022-01-24-kblab-publishes-an-article-in-crl/">https://kb-labb.github.io/posts/2022-01-24-kblab-publishes-an-article-in-crl/</a>.
</div></div></section></div> ]]></description>
  <guid>https://kb-labb.github.io/posts/2022-01-24-kblab-publishes-an-article-in-crl/</guid>
  <pubDate>Sun, 23 Jan 2022 23:00:00 GMT</pubDate>
  <media:content url="https://kb-labb.github.io/posts/2022-01-24-kblab-publishes-an-article-in-crl/newspapers.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>KBLab’s director Love Börjeson nominated as AI Swede of the year 2021</title>
  <dc:creator>Elena Fano</dc:creator>
  <link>https://kb-labb.github.io/posts/2021-11-15-kblabs-director-love-brjeson-nominated-as-ai-swede-of-the-year-2021/</link>
  <description><![CDATA[ 





<section id="about-techsverige" class="level3">
<h3 class="anchored" data-anchor-id="about-techsverige">About TechSverige</h3>
<p>From their own <a href="https://www.techsverige.se/">website</a>:<br>
“TechSverige is a member organization for companies of all sizes within the tech sector, that wish to join the largest industry network in Sweden in order to promote and further develop the tech market and conditions for tech enterprises. We represent about 1 400 member companies that between them have nearly 100 000 employees.”</p>
</section>
<section id="about-the-prize" class="level3">
<h3 class="anchored" data-anchor-id="about-the-prize">About the prize</h3>
<p>Årets AI Svensk 2021 (AI Swede of the year 2021) is a prize awarded by TechSverige. The prize aims to promote efforts and achievements that lead to an advancement in Swedish AI. That is done by way of bringing forward and honoring people who have contributed to the development and visibility of Swedish AI.</p>
</section>
<section id="motivation" class="level3">
<h3 class="anchored" data-anchor-id="motivation">Motivation</h3>
<p>“Love Börjeson leads KBLab at the National Library of Sweden with great patience and engagement. Among other things, the lab builds large AI language models for Swedish and has a huge impact in the application of AI in the Swedish public sector. KBLab is considered to be a unique resource in the development of Swedish AI and thanks to Love’s leadership and willpower it even has the potential to position Sweden on the international AI map.”</p>
<p>Read more in the official announcement <a href="https://www.techsverige.se/2021/11/har-ar-finalisterna-i-arets-ai-svensk-2021/">here</a> (in Swedish).</p>


</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{fano2021,
  author = {Fano, Elena},
  title = {KBLab’s Director {Love} {Börjeson} Nominated as {AI} {Swede}
    of the Year 2021},
  date = {2021-11-15},
  url = {https://kb-labb.github.io/posts/2021-11-15-kblabs-director-love-brjeson-nominated-as-ai-swede-of-the-year-2021/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-fano2021" class="csl-entry quarto-appendix-citeas">
Fano, Elena. 2021. <span>“KBLab’s Director Love Börjeson Nominated as AI
Swede of the Year 2021.”</span> November 15, 2021. <a href="https://kb-labb.github.io/posts/2021-11-15-kblabs-director-love-brjeson-nominated-as-ai-swede-of-the-year-2021/">https://kb-labb.github.io/posts/2021-11-15-kblabs-director-love-brjeson-nominated-as-ai-swede-of-the-year-2021/</a>.
</div></div></section></div> ]]></description>
  <guid>https://kb-labb.github.io/posts/2021-11-15-kblabs-director-love-brjeson-nominated-as-ai-swede-of-the-year-2021/</guid>
  <pubDate>Sun, 14 Nov 2021 23:00:00 GMT</pubDate>
  <media:content url="https://kb-labb.github.io/posts/2021-11-15-kblabs-director-love-brjeson-nominated-as-ai-swede-of-the-year-2021/ai_swede.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Introducing a Swedish Sentence Transformer</title>
  <dc:creator>Faton Rekathati</dc:creator>
  <link>https://kb-labb.github.io/posts/2021-08-23-a-swedish-sentence-transformer/</link>
  <description><![CDATA[ 





<p>Sentence transformers are a useful class of models that make it easier to implement efficient textual search applications. In this article we explain how KBLab’s Swedish Sentence-BERT was trained, providing some motivations on the methods used and the training process.</p>
<p>The published model can be found on Huggingface via the following link: <a href="https://huggingface.co/KBLab/sentence-bert-swedish-cased">https://huggingface.co/KBLab/sentence-bert-swedish-cased</a> .</p>
<section id="meaningful-sentence-embeddings" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="meaningful-sentence-embeddings">Meaningful sentence embeddings</h2>
<p>Pre-trained transformer language models trained at scale on large amounts of data have shown great success when fine-tuned on tasks such as text classification, named entity recognition and question answering. They can in fact also be successfully fine-tuned to compare the similarity between <em>two</em> sentences. However, when trained on semantic textual similarity (STS) tasks, these models typically require that two sentences be passed together as one input sequence to the network. This convention of passing sentence pairs as single input sequences is also present in the pre-training of said models, where one of the network’s pre-training tasks commonly includes “next sentence prediction” .</p>
<p>While this convention produces strong results because the model can draw and combine information from both sentences in solving a task, it unfortunately also leads to practical issues when a particular dataset does not come nicely arranged in the form of sentence pairs, but rather instead as an unordered set of sentences. Finding the top <img src="https://latex.codecogs.com/png.latex?k"> most similar sentences in a set of <img src="https://latex.codecogs.com/png.latex?N"> sentences requires <img src="https://latex.codecogs.com/png.latex?%5Csum%5EN_%7Bi=1%7D%20i%20=%20%5Cfrac%7BN%20%5Ccdot%20(N-1)%7D%7B2%7D"> similarity computations <span class="citation" data-cites="sentence-bert">(Reimers and Gurevych 2019)</span>. In the case of BERT, every single one of these computations come with an additional overhead – since in order to obtain a similarity score one must first pass every sentence pair through the neural network. A BERT base model consists of <img src="https://latex.codecogs.com/png.latex?110"> million parameters. These parameters are all involved in the transformation of the input to obtain a single similarity score.</p>

<div class="no-row-height column-margin column-container"><div class="">
<p><br> <br> <br> <br> A list of 2000 sentences requires almost 1 million similarity computations.</p>
</div></div><div id="fig-sbert" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-sbert-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="https://kb-labb.github.io/posts/2021-08-23-a-swedish-sentence-transformer/images/Bi_vs_Cross-Encoder.png" class="img-fluid figure-img">
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-sbert-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;1: Sentence similarity models were traditionally trained as cross-encoders (<strong>right figure</strong>). Information from Sentence A and Sentence B became "cross-encoded" , since both were passed as input together to the model. While effective at producing a similarity score, this setup would not yield meaningful sentence embeddings for the respective sentences A or B in isolation. This "cross-contamination" was addressed by <span class="citation" data-cites="sentence-bert">Reimers and Gurevych (2019)</span> using two BERT models in the training process, passing only one sentence each to the models (<strong>left figure</strong>), producing unique embeddings <strong>u</strong> and <strong>v</strong> for the respective sentences. <sup>1</sup>
</figcaption>
</figure>
</div>
<p><span class="citation" data-cites="sentence-bert">Reimers and Gurevych (2019)</span> estimated it would take approximately 65 hours to perform the required 50 million similarity inference computations for a set of 10000 sentences with a V100 GPU. As an alternative they proposed Sentence-BERT, which they gave the moniker a “bi-encoder” . In the training process each sentence was passed independently to the model. The resulting sentence embeddings <strong>u</strong> and <strong>v</strong> were then used for different down stream tasks such as classification. Trained in this manner, BERT models are successfully able to produce meaningful semantic sentence embeddings for single sentences.</p>
<p>Once the embeddings are obtained, we can perform the <img src="https://latex.codecogs.com/png.latex?%5Cfrac%7BN%20%5Ccdot%20(N-1)%7D%7B2%7D"> similarity computations using only the embeddings and consequently avoid the overhead of involving the neural network in each of the computations. In this scenario we only need <img src="https://latex.codecogs.com/png.latex?N"> inference passes through the network to obtain sentence embeddings for every sentence. As a result our inference time can be reduced from 65 hours to mere seconds.</p>
</section>
<section id="swedish-training-data" class="level2">
<h2 class="anchored" data-anchor-id="swedish-training-data">Swedish training data?</h2>
<p>An ever-present concern when training Swedish language models tends to be the lack of training data for fine-tuning. The situation is no different in the area of semantic textual similarity, where training data is plentiful in English, but sorely lacking in Swedish. Below is an example of data sources used to train many of the English sentence transformers models in the <a href="https://www.sbert.net/docs/installation.html"><code>sentence-transformers</code></a> package <sup>2</sup> . The evaluation dataset most often used is STSb: the Semantic Textual Similarity benchmark <span class="citation" data-cites="stsb">(Cer et al. 2017)</span>. A subset of the STSb’s test set was translated into Swedish and included as part of SuperLim <span class="citation" data-cites="superlim">(Adesam, Berdicevskis, and Morger 2020)</span> under the name of <a href="https://spraakbanken.gu.se/en/resources/sweparaphrase">SweParaphrase</a>.</p>
<table class="docutils" border="1">
<thead>
<tr>
<th>
Name
</th>
<th>
Source
</th>
<th align="center">
#Sentence-Pairs
</th>
<th align="center">
STSb-dev
</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<a href="https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/paraphrases/AllNLI.tsv.gz">AllNLI.tsv.gz</a>
</td>
<td>
<a href="https://nlp.stanford.edu/projects/snli/">SNLI</a> + <a href="https://cims.nyu.edu/~sbowman/multinli/">MultiNLI</a>
</td>
<td align="center">
277,230
</td>
<td align="center">
86.54
</td>
</tr>
<tr>
<td>
<a href="https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/paraphrases/sentence-compression.tsv.gz">sentence-compression.tsv.gz</a>
</td>
<td>
<a href="https://github.com/google-research-datasets/sentence-compression">sentence-compression</a>
</td>
<td align="center">
180,000
</td>
<td align="center">
84.36
</td>
</tr>
<tr>
<td>
<a href="https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/paraphrases/SimpleWiki.tsv.gz">SimpleWiki.tsv.gz</a>
</td>
<td>
<a href="https://cs.pomona.edu/~dkauchak/simplification/">SimpleWiki</a>
</td>
<td align="center">
102,225
</td>
<td align="center">
84.26
</td>
</tr>
<tr>
<td>
<a href="https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/paraphrases/altlex.tsv.gz">altlex.tsv.gz</a>
</td>
<td>
<a href="https://github.com/chridey/altlex/">altlex</a>
</td>
<td align="center">
112,696
</td>
<td align="center">
83.34
</td>
</tr>
<tr>
<td>
<a href="https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/paraphrases/msmarco-triplets.tsv.gz">msmarco-triplets.tsv.gz</a>
</td>
<td>
<a href="https://microsoft.github.io/msmarco/">MS MARCO Passages</a>
</td>
<td align="center">
5,028,051
</td>
<td align="center">
83.12
</td>
</tr>
<tr>
<td>
<a href="https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/paraphrases/quora_duplicates.tsv.gz">quora_duplicates.tsv.gz</a>
</td>
<td>
<a href="https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs">Quora</a>
</td>
<td align="center">
103,663
</td>
<td align="center">
82.55
</td>
</tr>
<tr>
<td>
<a href="https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/paraphrases/coco_captions-with-guid.tsv.gz">coco_captions-with-guid.tsv.gz</a>
</td>
<td>
<a href="https://cocodataset.org/">COCO</a>
</td>
<td align="center">
828,395
</td>
<td align="center">
82.25
</td>
</tr>
<tr>
<td>
<a href="https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/paraphrases/flickr30k_captions-with-guid.tsv.gz">flickr30k_captions-with-guid.tsv.gz</a>
</td>
<td>
<a href="https://shannon.cs.illinois.edu/DenotationGraph/">Flickr 30k</a>
</td>
<td align="center">
317,695
</td>
<td align="center">
82.04
</td>
</tr>
<tr>
<td>
<a href="https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/paraphrases/yahoo_answers_title_question.tsv.gz">yahoo_answers_title_question.tsv.gz</a>
</td>
<td>
<a href="https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset">Yahoo Answers Dataset</a>
</td>
<td align="center">
659,896
</td>
<td align="center">
81.19
</td>
</tr>
<tr>
<td>
<a href="https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/paraphrases/S2ORC_citation_pairs.tsv.gz">S2ORC_citation_pairs.tsv.gz</a>
</td>
<td>
<a href="http://s2-public-api-prod.us-west-2.elasticbeanstalk.com/corpus/">Semantic Scholar Open Research Corpus</a>
</td>
<td align="center">
52,603,982
</td>
<td align="center">
81.02
</td>
</tr>
<tr>
<td>
<a href="https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/paraphrases/yahoo_answerstitle_answer.tsv.gz">yahoo_answers_title_answer.tsv.gz</a>
</td>
<td>
<a href="https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset">Yahoo Answers Dataset</a>
</td>
<td align="center">
1,198,260
</td>
<td align="center">
80.25
</td>
</tr>
<tr>
<td>
<a href="https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/paraphrases/stackexchange_duplicate_questions.tsv.gz">stackexchange_duplicate_questions.tsv.gz</a>
</td>
<td>
<a href="https://stackexchange.com/">Stackexchange</a>
</td>
<td align="center">
169,438
</td>
<td align="center">
80.37
</td>
</tr>
<tr>
<td>
<a href="https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/paraphrases/yahoo_answers_question_answer.tsv.gz">yahoo_answers_question_answer.tsv.gz</a>
</td>
<td>
<a href="https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset">Yahoo Answers Dataset</a>
</td>
<td align="center">
681,164
</td>
<td align="center">
79.88
</td>
</tr>
<tr>
<td>
<a href="https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/paraphrases/wiki-atomic-edits.tsv.gz">wiki-atomic-edits.tsv.gz</a>
</td>
<td>
<a href="https://github.com/google-research-datasets/wiki-atomic-edits">wiki-atomic-edits</a>
</td>
<td align="center">
22,980,185
</td>
<td align="center">
79.58
</td>
</tr>
<tr>
<td>
<a href="https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/paraphrases/wiki-split.tsv.gz">wiki-split.tsv.gz</a>
</td>
<td>
<a href="https://github.com/google-research-datasets/wiki-split">wiki-split</a>
</td>
<td align="center">
929,944
</td>
<td align="center">
76.59
</td>
</tr>
</tbody>
</table>
<section id="machine-translation" class="level3">
<h3 class="anchored" data-anchor-id="machine-translation">Machine translation</h3>
<p>Different ways of getting around the issue of data have been explored. <span class="citation" data-cites="isbister-simply">Isbister and Sahlgren (2020)</span> investigated whether simply machine translating the English NLI and STS training datasets to Swedish could yield competitive results. They used <a href="https://huggingface.co/KB/bert-base-swedish-cased">KB-BERT</a> <span class="citation" data-cites="swedish-bert">(Malmsten, Börjeson, and Haffenden 2020)</span> trained in a cross-encoder setting and found it outperformed all other evaluated options (<img src="https://latex.codecogs.com/png.latex?82.5"> Pearson correlation on a machine translated version of STS-b test set). Ultimately the authors still recommended against using their model “due to a high prevalence of translation errors” in the data with unknown effects on downstream applications.</p>
</section>
<section id="self-supervised" class="level3">
<h3 class="anchored" data-anchor-id="self-supervised">Self-supervised</h3>
<p>A second avenue for getting around the lack of training data has been by training completely self-supervised/unsupervised. <span class="citation" data-cites="carlsson">Carlsson et al. (2021)</span> trained their models on data dumps of Wikipedia. Contrastive tension was used to maximize similarity between identical sentences, and minimize it for differing sentences without any need for labels. Their model however performed somewhat worse for Swedish than other languages (Arabic, English, Russian, Spanish). The Swedish model achieved <img src="https://latex.codecogs.com/png.latex?61.69"> Pearson correlation on the machine translated version of the STS-b test set created by <span class="citation" data-cites="isbister-simply">Isbister and Sahlgren (2020)</span> .</p>
<p>While the results are encouraging, at this point there still existed a gap that had yet to be closed when compared to the best performing English models.</p>
</section>
</section>
<section id="data-parallel-corpus-training-data" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="data-parallel-corpus-training-data">Data: Parallel corpus training data</h2>
<p>One approach to training a model is through distilling the knowledge of a “teacher” model to a student model. In this scenario where we want our student model to emulate the embeddings of a teacher model trained in another language, we need to make use of parallel (translated) data.</p>
<p>We have data in the form of one sentence from the source language (English) and one sentence from the target language (Swedish). An example of the first 100 sentences of the English-Swedish Europarl dataset can be found <a href="https://opus.nlpl.eu/Europarl/v8/en-sv_sample.html">here</a>.</p>
<p>The OPUS (Open Parallel Corpus) project <span class="citation" data-cites="opus">(Tiedemann 2012)</span> has made available a great collection of parallel corpora from the web from a diverse set of sources. We use a number of different data sources to ensure our translated parallel sentences cover a wide range of language. Below is a table of the number of total sentences from each dataset used in the training of our model. Each English and Swedish sentence is counted uniquely. The reason some datasets have an odd number of sentences, is because in rare occasions two (or more) different candidate translations may be attached to the same sentence. We filter out all sentences above 600 characters in length. Furthermore, single sentences without a suggested candidate translation or source sentence are not included in training.</p>
<div class="cell">
<div class="cell-output-display">
<div class="reactable html-widget html-fill-item" id="htmlwidget-8473f67a064cebc3b2cb" style="width:auto;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-8473f67a064cebc3b2cb">{"x":{"tag":{"name":"Reactable","attribs":{"data":{"Dataset":["OpenSubtitles 18","Europarl","JW300","EMEA","EUbookshop","TED2020","Tatoeba"],"Sentences_v1":[6360393,3680593,3248845,651402,355341,232643,51972],"Sentences_v2":["TBD","TBD","TBD","TBD","TBD","TBD","TBD"]},"columns":[{"id":"Dataset","name":"Dataset","type":"character"},{"id":"Sentences_v1","name":"Sentences_v1","type":"numeric"},{"id":"Sentences_v2","name":"Sentences_v2","type":"character"}],"highlight":true,"dataKey":"d26c7fe883712464382930a3af90d898"},"children":[]},"class":"reactR_markup"},"evals":[],"jsHooks":[]}</script>
</div>
</div>
<p>The question of dataset bias may naturally arise as a result of our selection. However, it should be noted that we – in our training – are merely recreating and matching the knowledge of the original model. The training process is not meant to impart “new knowledge” to our student model. Biases in our datasets are unlikely to be reflected in any substantial manner in the final model. Rather the final model will reflect the biases already present in the teacher model.</p>
<p>Here is a short description of each dataset:</p>
<ul>
<li><a href="https://opus.nlpl.eu/OpenSubtitles-v2018.php"><strong>OpenSubtitles 18</strong></a>: Sourced from <a href="http://www.opensubtitles.org/">opensubtitles.org</a>, a large database of movie and TV subtitles <span class="citation" data-cites="opensubtitles">(Lison and Tiedemann 2016)</span>. Can be filtered by a variable that measures time-overlap of the subtitles. v1 of the model did not filter observations, but a future v2 of the model will be trained on a subset of data with a higher <code>overlap</code> threshold.</li>
<li><a href="https://opus.nlpl.eu/Europarl.php"><strong>Europarl</strong></a>: A parallel corpus consisting of proceedings of the European parliament <span class="citation" data-cites="opus">(Tiedemann 2012)</span>.</li>
<li><a href="https://opus.nlpl.eu/JW300.php"><strong>JW300</strong></a>: Various texts and articles from different Jehovas Witnesses websites and magazines. A filtering variable denoting <code>certainty</code> of alignment exists. Filtering will be applied for v2 of the model.</li>
<li><a href="https://opus.nlpl.eu/EMEA.php"><strong>EMEA</strong></a>: Texts from the European Medicines Agency, extracted from PDFs.</li>
<li><a href="https://opus.nlpl.eu/EUbookshop.php"><strong>EUbookshop</strong></a>: Based on documents from the EU bookshop. This datasets xml alignment file was found to be corrupt after training the model, causing quality issues in the aligned sentences. Future versions of the model will likely omit this dataset unless the corrupted xml files can be fixed.</li>
<li><a href="https://opus.nlpl.eu/TED2020.php"><strong>TED2020</strong></a>: Based off of TED and TED-X transcripts from July 2020 <span class="citation" data-cites="multilingual-sentence-bert">(Reimers and Gurevych 2020)</span>.</li>
<li><a href="https://opus.nlpl.eu/TED2020.php"><strong>Tatoeba</strong></a>: Translated sentences from a free collaborative platform for language learners.</li>
</ul>

<div class="no-row-height column-margin column-container"><div class="">
<p>Data download scripts can be found on the following links: <a href="https://github.com/kb-labb/swedish-sbert/blob/main/get_parallel_data_opus.py"><code>get_parallel_data_opus.py</code></a> <a href="https://github.com/kb-labb/swedish-sbert/blob/main/get_parallel_data_tatoeba.py"><code>get_parallel_data_tatoeba.py</code></a> <a href="https://github.com/kb-labb/swedish-sbert/blob/main/get_parallel_data_ted2020.py"><code>get_parallel_data_ted2020.py</code></a></p>
</div></div><section id="dev-set-validation-during-training" class="level3">
<h3 class="anchored" data-anchor-id="dev-set-validation-during-training">Dev set validation during training</h3>
<p>We split off 1000 sentence pairs each from TED2020 and Tatoeba to form a validation set. The model was validated against these every 1000 training steps. The best model was continually autosaved during training based on the lowest combined MSE (sum) of the student model’s English and Swedish sentence embeddings against the teacher model’s English sentence embedding.</p>
</section>
</section>
<section id="method-translating-models-via-knowledge-distillation" class="level2">
<h2 class="anchored" data-anchor-id="method-translating-models-via-knowledge-distillation">Method: Translating models via knowledge distillation</h2>
<p>The method used for KBLab’s Sentence-BERT is described in the paper “Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation” <span class="citation" data-cites="multilingual-sentence-bert">(Reimers and Gurevych 2020)</span>. In short it allows us to take existing sentence embeddings models of the Bi-Encoder type shown in figure @ref(fig:cross) and extend them to new languages. The general setup in our case is as follows:</p>
<ul>
<li>A strong pre-trained teacher model of the Bi-Encoder type maps sentences in a source language (English) to dense vectors (embeddings). Our goal is to make a student model learn to match the teacher’s embeddings.<br>
</li>
<li>A student model, which may be of the Cross-Encoder type takes sentence pairs consisting of <em>one</em> sentence from the <em>source language</em> (English) and <em>one</em> sentence from the <em>target language</em> (Swedish). Its objective is to minimize the mean squared error (MSE) between the teacher’s embedding against both the source and target language embeddings generated by the student model. See figure @ref(fig:multilingual).</li>
<li>The input sentences to the student model are from parallel corpora, meaning they are translations of each other.</li>
</ul>
<div id="fig-multilingual" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-multilingual-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="https://kb-labb.github.io/posts/2021-08-23-a-swedish-sentence-transformer/images/multilingual.jpg" class="img-fluid figure-img">
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-multilingual-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<strong>Figure</strong>&nbsp;2: Student model takes a sentence from a source language and a sentence from a target language and minimizes the mean squared error of each to the teacher model’s embedding. Image source from <span class="citation" data-cites="multilingual-sentence-bert">Reimers and Gurevych (2020)</span>.
</figcaption>
</figure>
</div>
<p><strong><em>Update 2022-04-22:</em></strong> In a previous version of this post the author discussed and alluded to the English-Swedish input sentence pairs being cross-encoded in the student model. However, when training a bi-encoder sentence transformer the sentences from different languages are processed independently by the student model network.</p>
<section id="teacher-model" class="level3">
<h3 class="anchored" data-anchor-id="teacher-model">Teacher model</h3>
<p>We chose <a href="https://www.sbert.net/docs/pretrained_models.html#sentence-embedding-models"><code>paraphrase-mpnet-base-v2</code></a> as our teacher model. At the time of training this was the strongest available bi-encoder. It was trained on <a href="https://www.sbert.net/examples/training/paraphrases/README.html">Paraphrase Data</a>.</p>
</section>
<section id="student-model" class="level3">
<h3 class="anchored" data-anchor-id="student-model">Student model</h3>
<p>Our student model was the Swedish pretrained <a href="https://huggingface.co/KB/bert-base-swedish-cased">KB-BERT</a>, using the same vocabulary.</p>
</section>
</section>
<section id="evaluations-on-superlim" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="evaluations-on-superlim">Evaluations on SuperLim</h2>
<p>SuperLim is a Swedish evaluation suite for natural language understanding models <span class="citation" data-cites="superlim">(Adesam, Berdicevskis, and Morger 2020)</span>. It was inspired by the English SuperGLUE <span class="citation" data-cites="superglue">(Wang et al. 2020)</span>.</p>
<p>We chose the four resources most relevant to the tasks our model was trained on:</p>
<ul>
<li><p><a href="https://spraakbanken.gu.se/en/resources/sweparaphrase">SweParaphrase</a>: A subset of the English STS benchmark <span class="citation" data-cites="stsb">(Cer et al. 2017)</span> dataset translated to Swedish. It consists of 165 sentence pairs. Human evaluators ranked the sentences according to how similar the two sentences were deemed to be (from 0 meaning no meaning overlap, to 5 meaning equivalence). Assembled and translated to Swedish by <span class="citation" data-cites="isbister-simply">Isbister and Sahlgren (2020)</span>. <strong><em>Update (correction, 2021-09-07):</em></strong> The English STSb was machine translated to Swedish by <span class="citation" data-cites="isbister-simply">Isbister and Sahlgren (2020)</span>. A subset of 165 sentence pairs from this automatically translated dataset were manually corrected by a native speaker of Swedish as part of the SuperLim project.</p></li>
<li><p><a href="https://spraakbanken.gu.se/resurser/faq">Swedish FAQ</a>: A collection of questions and answers from various Swedish authorities websites (Försäkringskassan, Skatteverket, etc). The questions are divided into categories, for example <code>Förälder :: Barnbidrag :: Vanliga frågor</code>. The task is to match questions within a category to the correct answer (among a set of candidate answers that have been shuffled within the category).</p></li>
<li><p><a href="https://spraakbanken.gu.se/en/resources/swesat-synonyms">SweSAT synonyms</a>: Multiple choice word synonym task of Högskoleprovet (Swedish equivalent to SAT). Test taker is presented with a question word, and needs to match it to the correct option (synonym) from 5 possible choices.</p></li>
<li><p><a href="https://spraakbanken.gu.se/en/resources/supersim-superlim">SuperSim</a>: It is not clear whether Sentence embedding models produce meaningful word embeddings. This is a similarity and relatedness test set for word pairs, where each word has been rated on both relatedness and similarity by five different annotators. We only evaluate against the similarity scores.</p></li>
</ul>
<section id="sweparaphrase-results" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="sweparaphrase-results">SweParaphrase results</h3>
<p>We compare our results (KB-SBERT) to <del>the previous highest published scores on SweParaphrase reported by</del> <span class="citation" data-cites="isbister-simply">Isbister and Sahlgren (2020)</span>. Results reported as <code>Correlation coefficient * 100</code>.</p>
<p><strong>Update (article correction, 2021-09-07):</strong> <span class="citation" data-cites="isbister-simply">Isbister and Sahlgren (2020)</span> evaluated on a machine translated version of the STSb test set. SweParaphrase is a subset of this machine translated version consisting of 165 sentence pairs. The sentence pairs were manually corrected by a native Swedish speaker. The results are therefore not directly comparable.</p>
<div class="cell">
<div class="cell-output-display">
<div class="reactable html-widget html-fill-item" id="htmlwidget-2ad3d7f35d4d4bc50368" style="width:auto;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-2ad3d7f35d4d4bc50368">{"x":{"tag":{"name":"Reactable","attribs":{"data":{"Model":["KB-BERT (Isbister & Sahlgren)","KB-SBERT"],"Pearson":[82.5,91.83],"Spearman":["NA",91.14]},"columns":[{"id":"Model","name":"Model","type":"character"},{"id":"Pearson","name":"Pearson","type":"numeric"},{"id":"Spearman","name":"Spearman","type":"numeric"}],"highlight":true,"dataKey":"9d3273764cad63ba4268d3293355d9e9"},"children":[]},"class":"reactR_markup"},"evals":[],"jsHooks":[]}</script>
</div>
</div>

<div class="no-row-height column-margin column-container"><div class="">
<p>Code to replicate results: <a href="https://github.com/kb-labb/swedish-sbert/blob/main/evaluate_sweparaphrase.py">evaluate_sweparaphrase.py</a></p>
</div></div></section>
<section id="swedish-faq-results" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="swedish-faq-results">Swedish FAQ results</h3>
<p>There’s a varying number of questions per category in this dataset. Randomly guessing or naïvely guessing a single candidate answer within each group would give us an expected accuracy of <img src="https://latex.codecogs.com/png.latex?9.55%5C%25"> (average questions per category is <img src="https://latex.codecogs.com/png.latex?10.47">).</p>
<p>KB-SBERT total accuracy: <img src="https://latex.codecogs.com/png.latex?50.49%5C%25">.</p>

<div class="no-row-height column-margin column-container"><div class="">
<p>Replicate results: <a href="https://github.com/kb-labb/swedish-sbert/blob/main/evaluate_faq.py">evaluate_faq.py</a></p>
</div></div><p>KB-SBERT manages to match half of the questions with the correct answer.</p>
</section>
<section id="swesat-synonyms" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="swesat-synonyms">SweSAT synonyms</h3>
<p>KB-SBERT hasn’t been explicitly trained to generate meaningful embeddings on the word level. However, we are curious to see how the model performs. Randomly guessing a single answer alternative would yield an expected accuracy of <img src="https://latex.codecogs.com/png.latex?20%5C%25">.</p>
<p>KB-SBERT accuracy: <img src="https://latex.codecogs.com/png.latex?42.82%5C%25">.</p>

<div class="no-row-height column-margin column-container"><div class="">
<p>Replicate results: <a href="https://github.com/kb-labb/swedish-sbert/blob/main/evaluate_swesat.py">evaluate_swesat.py</a></p>
</div></div><p>The model performs better than random, though the result is still quite weak.</p>
</section>
<section id="supersim" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="supersim">SuperSim</h3>
<p>Here we predict semantic word similarity between word pairs as opposed to sentence pairs. We compare results with baselines published by <span class="citation" data-cites="supersim">Hengchen and Tahmasebi (2021)</span> on the word similarity task.</p>
<div class="cell">
<div class="cell-output-display">
<div class="reactable html-widget html-fill-item" id="htmlwidget-512029f6f50bb0682c2b" style="width:auto;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-512029f6f50bb0682c2b">{"x":{"tag":{"name":"Reactable","attribs":{"data":{"Model":["fastText","GloVe","Word2Vec","KB-SBERT"],"Spearman":[52.8,49.9,49.6,34.3]},"columns":[{"id":"Model","name":"Model","type":"character"},{"id":"Spearman","name":"Spearman","type":"numeric"}],"highlight":true,"dataKey":"ce97e883105b0cded05007cb90f72261"},"children":[]},"class":"reactR_markup"},"evals":[],"jsHooks":[]}</script>
</div>
</div>

<div class="no-row-height column-margin column-container"><div class="">
<p>Replicate results: <a href="https://github.com/kb-labb/swedish-sbert/blob/main/evaluate_supersim.py">evaluate_supersim.py</a></p>
</div></div><p>It appears KB-SBERT is not to recommend for word embeddings.</p>
</section>
</section>
<section id="acknowledgements" class="level2">
<h2 class="anchored" data-anchor-id="acknowledgements">Acknowledgements</h2>
<p>We gratefully acknowledge the HPC RIVR consortium (<a href="www.hpc-rivr.si">www.hpc-rivr.si</a>) and EuroHPC JU (<a href="eurohpc-ju.europa.eu">eurohpc-ju.europa.eu</a>) for funding this research by providing computing resources of the HPC system Vega at the Institute of Information Science (<a href="www.izum.si">www.izum.si</a>).</p>
</section>
<section id="closing-words" class="level2">
<h2 class="anchored" data-anchor-id="closing-words">Closing words</h2>
<p>KB-SBERT appears to perform well on sentence similarity tasks. Training on parallel corpora using a teacher model seemingly leads to better results compared to machine translation. However, it remains to be seen whether the distilled model behaves well when fine-tuned on downstream tasks in Swedish without any English supervision.</p>
<p>If you use KB-SBERT in your work, and perhaps fine-tune it for specific tasks, please drop us a message and tell us how it went. You can find the lab’s e-mail address in the footer of this webpage.</p>



</section>


<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-bibliography"><h2 class="anchored quarto-appendix-heading">References</h2><div id="refs" class="references csl-bib-body hanging-indent" data-entry-spacing="0">
<div id="ref-superlim" class="csl-entry">
Adesam, Yvonne, Aleksandrs Berdicevskis, and Felix Morger. 2020. <span>“SwedishGLUE – Towards a Swedish Test Set for Evaluating Natural Language Understanding Models.”</span>
</div>
<div id="ref-carlsson" class="csl-entry">
Carlsson, Fredrik, Evangelina Gogoulou, Erik Ylipää, Amaru Cuba Gyllensten, and Magnus Sahlgren. 2021. <em>Semantic Re-Tuning with Contrastive Tension</em>. International Conference on Learning Representations, <span>ICLR</span> 2021. <a href="https://openreview.net/pdf?id=Ov_sMNau-PF">https://openreview.net/pdf?id=Ov_sMNau-PF</a>.
</div>
<div id="ref-stsb" class="csl-entry">
Cer, Daniel, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. <span>“SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation.”</span> <em>Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)</em>. <a href="https://doi.org/10.18653/v1/s17-2001">https://doi.org/10.18653/v1/s17-2001</a>.
</div>
<div id="ref-supersim" class="csl-entry">
Hengchen, Simon, and Nina Tahmasebi. 2021. <span>“<span>S</span>uper<span>S</span>im: A Test Set for Word Similarity and Relatedness in <span>S</span>wedish.”</span> In <em>Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)</em>, 268–75. Reykjavik, Iceland (Online): Link<span>ö</span>ping University Electronic Press, Sweden. <a href="https://aclanthology.org/2021.nodalida-main.27">https://aclanthology.org/2021.nodalida-main.27</a>.
</div>
<div id="ref-isbister-simply" class="csl-entry">
Isbister, Tim, and Magnus Sahlgren. 2020. <span>“Why Not Simply Translate? A First Swedish Evaluation Benchmark for Semantic Similarity.”</span> <a href="https://arxiv.org/abs/2009.03116">https://arxiv.org/abs/2009.03116</a>.
</div>
<div id="ref-opensubtitles" class="csl-entry">
Lison, Pierre, and Jörg Tiedemann. 2016. <span>“<span>O</span>pen<span>S</span>ubtitles2016: Extracting Large Parallel Corpora from Movie and <span>TV</span> Subtitles.”</span> In <em>Proceedings of the Tenth International Conference on Language Resources and Evaluation (<span>LREC</span>’16)</em>, 923–29. Portoro<span>ž</span>, Slovenia: European Language Resources Association (ELRA). <a href="https://aclanthology.org/L16-1147">https://aclanthology.org/L16-1147</a>.
</div>
<div id="ref-swedish-bert" class="csl-entry">
Malmsten, Martin, Love Börjeson, and Chris Haffenden. 2020. <span>“Playing with Words at the National Library of Sweden – Making a Swedish BERT.”</span> <a href="https://arxiv.org/abs/2007.01658">https://arxiv.org/abs/2007.01658</a>.
</div>
<div id="ref-sentence-bert" class="csl-entry">
Reimers, Nils, and Iryna Gurevych. 2019. <span>“Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks.”</span> In <em>Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing</em>. Association for Computational Linguistics. <a href="https://arxiv.org/abs/1908.10084">https://arxiv.org/abs/1908.10084</a>.
</div>
<div id="ref-multilingual-sentence-bert" class="csl-entry">
———. 2020. <span>“Making Monolingual Sentence Embeddings Multilingual Using Knowledge Distillation.”</span> In <em>Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing</em>. Association for Computational Linguistics. <a href="https://arxiv.org/abs/2004.09813">https://arxiv.org/abs/2004.09813</a>.
</div>
<div id="ref-opus" class="csl-entry">
Tiedemann, Jörg. 2012. <span>“Parallel Data, Tools and Interfaces in OPUS.”</span> In <em>Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)</em>, edited by Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Mehmet Ugur Dogan, Bente Maegaard, Joseph Mariani, Jan Odijk, and Stelios Piperidis. Istanbul, Turkey: European Language Resources Association (ELRA).
</div>
<div id="ref-superglue" class="csl-entry">
Wang, Alex, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2020. <span>“SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems.”</span> <a href="https://arxiv.org/abs/1905.00537">https://arxiv.org/abs/1905.00537</a>.
</div>
</div></section><section id="footnotes" class="footnotes footnotes-end-of-document"><h2 class="anchored quarto-appendix-heading">Footnotes</h2>

<ol>
<li id="fn1"><p>Image source: https://www.sbert.net/docs/pretrained_cross-encoders.html↩︎</p></li>
<li id="fn2"><p>Paraphrase datasets table source: https://www.sbert.net/examples/training/paraphrases/README.html#↩︎</p></li>
</ol>
</section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{rekathati2021,
  author = {Rekathati, Faton},
  title = {Introducing a {Swedish} {Sentence} {Transformer}},
  date = {2021-08-23},
  url = {https://kb-labb.github.io/posts/2021-08-23-a-swedish-sentence-transformer/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-rekathati2021" class="csl-entry quarto-appendix-citeas">
Rekathati, Faton. 2021. <span>“Introducing a Swedish Sentence
Transformer.”</span> August 23, 2021. <a href="https://kb-labb.github.io/posts/2021-08-23-a-swedish-sentence-transformer/">https://kb-labb.github.io/posts/2021-08-23-a-swedish-sentence-transformer/</a>.
</div></div></section></div> ]]></description>
  <guid>https://kb-labb.github.io/posts/2021-08-23-a-swedish-sentence-transformer/</guid>
  <pubDate>Sun, 22 Aug 2021 22:00:00 GMT</pubDate>
</item>
</channel>
</rss>