Commit b008d281 authored by Fabio Frazao's avatar Fabio Frazao
Browse files

create_database_simpler->Fix typos on tutorial

parent 33ecd638
......@@ -13078,11 +13078,11 @@ div#notebook {
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Creating-a-training-database">Creating a training database<a class="anchor-link" href="#Creating-a-training-database">&#182;</a></h1><p>In this tutorial, we will use Ketos to create a training database that can be used to train a deep learning-based classifier.</p>
<p>We will use a subset of the data described in Kirsebom et al. 2020. It consists of 3s long clips, some containing right whale upcalls and others containing background noise. The clips are wavefiles extracted from recordings produced by bottom-mounted hydrophones in the Gulf of Saint Lawrence, Canada.</p>
<p>Our starting point will be a collection of wav files accompanied by annotations. You can find them in the ''data'' folder within the .zip file linked at the top of this page.There are In the train folder, there are 2000 files, half of them containing upcalls and the other half containing background noise (which, for our purpose, is anything that does not contain upcalls and might include sounds produced by other animals and the overall ambient noise). The annotations_train.csv file contains the label attributed to each file: 1 for upcall, 0 for background. Similarly, the val (validation) folder contains 200 .wav files (50% with upcalls) and is also accompanied by a annotations_test.csv file.</p>
<p>We will use a subset of the data described in <a href="https://asa.scitation.org/doi/10.1121/10.0001132">Kirsebom et al. 2020</a>. It consists of 3s long clips, some containing right whale upcalls and others containing background noise. The clips are wavefiles extracted from recordings produced by bottom-mounted hydrophones in the Gulf of Saint Lawrence, Canada.</p>
<p>Our starting point will be a collection of .wav files accompanied by annotations. You can find them in the ''data'' folder within the .zip file linked at the top of this page.There are In the train folder, there are 2000 files, half of them containing upcalls and the other half containing background noise (which, for our purposes, is anything that does not contain upcalls and might include sounds produced by other animals and the overall ambient noise). The annotations_train.csv file contains the label attributed to each file: 1 for upcall, 0 for background. Similarly, the val (validation) folder contains 200 .wav files (50% with upcalls) and is accompanied by the annotations_test.csv file.</p>
<p>We will use ketos to produce a database with spectrogram representations of the train and validation clips, so that we can later train deep learning based classifier able to distinguish the upcalls from the other sounds. Eventually, we will use that classifier to build a detector.</p>
<p>A different scenario would be where you have audio recordings and annotations indicating where in these recording the signals of interest are, but you don't have uniform clips of your with examples of the target signal(s) and background. That case is covered in <a href="#">this tutorial</a> .</p>
<p>We also encourage you to explore the documentation, since Ketos has a variety of tools that might help you to build training databases.</p>
<p>A different scenario would be where you have audio recordings and annotations indicating where in these recording the signals of interest are, but you don't have uniform clips of with examples of the target signal(s) and background. That case is covered in <a href="https://docs.meridian.cs.dal.ca/ketos/tutorials/create_database/index.html">this tutorial</a>.</p>
<p>We also encourage you to explore the <a href="https://docs.meridian.cs.dal.ca/ketos/index.html">documentation</a>, since Ketos has a variety of tools that might help you to build training databases in different scenarios.</p>
<h2 id="Contents:">Contents:<a class="anchor-link" href="#Contents:">&#182;</a></h2><p><a href="#section1">1. Importing the packages</a><br>
<a href="#section2">2. Loading the annotations</a><br>
<a href="#section3">3. Putting the annotations in the Ketos format</a><br>
......@@ -13896,7 +13896,7 @@ div#notebook {
<p>The <strong>annot_train</strong> dataframe contains 2000 rows and the <strong>annot_val</strong> 200.
The columns indicate:</p>
<p><strong>sound_file:</strong> name of the audio file<br>
<strong>label:</strong> label for the annotation (in our case, all annotated signals are 'upcalls', but the origincal DCLDE2013 dataset also had 'gunshots')</p>
<strong>label:</strong> label for the annotation (1 for upcall, 0 for background))</p>
</div>
</div>
......@@ -13913,7 +13913,7 @@ The columns indicate:</p>
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="3.-Putting-the-annotations-in-the-Ketos-format">3. Putting the annotations in the Ketos format<a class="anchor-link" href="#3.-Putting-the-annotations-in-the-Ketos-format">&#182;</a></h3><p>Let's check if our annotations follow the Ketos standard.</p>
<p>If that's the case, the function <code>sl.is_standardized</code> will return <code>False</code>.</p>
<p>If that's the case, the function <code>sl.is_standardized</code> will return <code>True</code>.</p>
</div>
</div>
......@@ -13934,6 +13934,49 @@ The columns indicate:</p>
<div class="output">
<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre> Your table is not in the Ketos format.
It should have two levels of indices: filename and annot_id.
It should also contain at least the &#39;label&#39; column.
If your annotations have time information, these should appear in the &#39;start&#39; and &#39;end&#39; columns
extra columns are allowed.
Here is a minimum example:
label
filename annot_id
file1.wav 0 2
1 1
2 2
file2.wav 0 2
1 2
2 1
And here is a table with time information and a few extra columns (&#39;min_freq&#39;, &#39;max_freq&#39; and &#39;file_time_stamp&#39;)
start end label min_freq max_freq file_time_stamp
filename annot_id
file1.wav 0 7.0 8.1 2 180.6 294.3 2019-02-24 13:15:00
1 8.5 12.5 1 174.2 258.7 2019-02-24 13:15:00
2 13.1 14.0 2 183.4 292.3 2019-02-24 13:15:00
file2.wav 0 2.2 3.1 2 148.8 286.6 2019-02-24 13:30:00
1 5.8 6.8 2 156.6 278.3 2019-02-24 13:30:00
2 9.0 13.0 1 178.2 304.5 2019-02-24 13:30:00
</pre>
</div>
</div>
<div class="output_area">
<div class="prompt output_prompt">Out[5]:</div>
......@@ -13956,7 +13999,7 @@ The columns indicate:</p>
<div class="prompt input_prompt">In&nbsp;[6]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">sl</span><span class="o">.</span><span class="n">is_standardized</span><span class="p">(</span><span class="n">annot_val</span><span class="p">)</span>
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">sl</span><span class="o">.</span><span class="n">is_standardized</span><span class="p">(</span><span class="n">annot_val</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</pre></div>
</div>
......@@ -13988,9 +14031,9 @@ The columns indicate:</p>
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Neither of our annotations are in the format ketos expects. But we can use the <code>sl.standardize</code> function to convert to the specified format.</p>
<p>The <em>annot_id</em> column is created automatically by the <code>sl.standardize</code> function. From the remaining required columns infdicated in the example above, we already have <em>start</em>, <em>end</em> and <em>label</em>. Our <em>sound_file</em> column needs to be renamed to <em>filename</em>, so we will need to provide a dictionary to specify that.</p>
<p>We have one extra column, <em>datetime</em>, that we don't really need to keep, so we'll set <code>trim_table=False</code>, which will only include a the required columns in the standardized tables.</p>
<p>If we wanted to keep the datetime (or any other columns), we would just set <code>trim_table=True</code>. One situation in which you might to do that is you need this information to split a dataset in train/test or train/validation/test, because then you can sort all your annotations by time and make sure the training training set does not overlap with the validation/test. But in our case, the annotations are already split.</p>
<p>The <em>annot_id</em> column is created automatically by the <code>sl.standardize</code> function. From the remaining required columns indicated in the example above, we already have <em>start</em>, <em>end</em> and <em>label</em>. Our <em>sound_file</em> column needs to be renamed to <em>filename</em>, so we will need to provide a dictionary to specify that.</p>
<p>We have one extra column, <em>datetime</em>, that we don't really need to keep, so we'll set <code>trim_table=True</code>, which will exclude any columns that are not required by the standardized tables.</p>
<p>If we wanted to keep the datetime (or any other columns), we would just set <code>trim_table=False</code>. One situation in which you might to do that is if you need this information to split a dataset in train/test or train/validation/test, because then you can sort all your annotations by time and make sure the training set does not overlap with the validation/test. But in our case, the annotations are already split.</p>
</div>
</div>
......@@ -14767,7 +14810,7 @@ The columns indicate:</p>
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="4.-Choosing-the-spectrogram-settings">4. Choosing the spectrogram settings<a class="anchor-link" href="#4.-Choosing-the-spectrogram-settings">&#182;</a></h3><p>As mentioned earlier, we'll represent the segments as spectrograms.
In the .zip file where you found the data, there's also a spectrogram configuration file (<code>spec_config.json</code>) which contains the settings we want to use.</p>
<p>This configuration file is simply a text file in the <code>.json</code> format, so you could make a copy of it, change a fee parameters and save several settings to use later or to share the with someone else.</p>
<p>This configuration file is simply a text file in the <code>.json</code> format, so you could make a copy of it, change a few parameters and save several settings to use later or to share the with someone else.</p>
</div>
</div>
......@@ -14865,9 +14908,9 @@ In the .zip file where you found the data, there's also a spectrogram configurat
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="5.-Creating-the-database">5. Creating the database<a class="anchor-link" href="#5.-Creating-the-database">&#182;</a></h3><p>Now, we have to compute the spectrograms following the setting above for each selection in our selection tables (i.e.: ach 3s clip) and then save them in a database.</p>
<h3 id="5.-Creating-the-database">5. Creating the database<a class="anchor-link" href="#5.-Creating-the-database">&#182;</a></h3><p>Now, we have to compute the spectrograms following the settings above for each selection in our selection tables (i.e.: each 3s clip) and then save them in a database.</p>
<p>All of this can be done with the <code>dbi.create_database</code> function in Ketos.</p>
<p>We will start with the training dataset. We need to indicate the name for the database we want to create, where the audio files are, a name for the dataset, the selections table and, finally the audio representation. As specified in our <code>spec_cfg</code>, this is a Magnitude spectrogram, but ketos can also create databases with Power, Mel and CQT spectrograms, as well as time-domain data (waveforms).</p>
<p>We will start with the training dataset. We need to indicate the name for the database we want to create, where the audio files are, a name for the dataset, the selections table and the audio representation. As specified in our <code>spec_cfg</code>, this is a Magnitude spectrogram, but ketos can also create databases with Power, Mel and CQT spectrograms, as well as time-domain data (waveforms).</p>
</div>
</div>
......@@ -14897,7 +14940,7 @@ In the .zip file where you found the data, there's also a spectrogram configurat
<div class="output_subarea output_stream output_stderr output_text">
<pre>100%|██████████| 2000/2000 [04:47&lt;00:00, 7.13it/s]</pre>
<pre>100%|██████████| 2000/2000 [00:16&lt;00:00, 124.81it/s]</pre>
</div>
</div>
......@@ -14960,7 +15003,7 @@ In the .zip file where you found the data, there's also a spectrogram configurat
<div class="output_subarea output_stream output_stderr output_text">
<pre>100%|██████████| 200/200 [00:28&lt;00:00, 6.99it/s]</pre>
<pre>100%|██████████| 200/200 [00:01&lt;00:00, 120.67it/s]</pre>
</div>
</div>
......@@ -15070,7 +15113,7 @@ In the .zip file where you found the data, there's also a spectrogram configurat
<div class="prompt input_prompt">In&nbsp;[17]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">db</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">db</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> <span class="c1">#Close the dataabse connection</span>
</pre></div>
</div>
......@@ -15089,7 +15132,7 @@ In the .zip file where you found the data, there's also a spectrogram configurat
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>You will likely not need to directly interact with the database. In a following tutorial, we will use Ketos to build a deep neural network and train it to recognize upcalls. Ketos handles the database interactions, so we won't really have to go into the details of it, but if you would like to learn more about how to get data from this database, take a look at the <a href="#">database_interface</a> module in ketos and the <a href="https://www.pytables.org/index.html">pyTables</a> documentation.</p>
<p>You will likely not need to directly interact with the database. In a following tutorial, we will use Ketos to build a deep neural network and train it to recognize upcalls. Ketos handles the database interactions, so we won't really have to go into the details of it, but if you would like to learn more about how to get data from this database, take a look at the <a href="https://docs.meridian.cs.dal.ca/ketos/modules/data_handling/database_interface.html">database_interface</a> module in ketos and the <a href="https://www.pytables.org/index.html">pyTables</a> documentation.</p>
</div>
</div>
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment