ketos issueshttps://gitlab.meridian.cs.dal.ca/public_projects/ketos/-/issues2021-11-05T15:26:27Zhttps://gitlab.meridian.cs.dal.ca/public_projects/ketos/-/issues/100Neural net calibration2021-11-05T15:26:27ZOliver KirsebomNeural net calibration[This paper](/uploads/e04565ff79df21019e3abf0127e50ad1/guo2017_calibrating_NN.pdf) describes a straightforward approach (Platt/Temperature scaling) to ``calibrating'' neural networks so that the output scores correspond more closely to p...[This paper](/uploads/e04565ff79df21019e3abf0127e50ad1/guo2017_calibrating_NN.pdf) describes a straightforward approach (Platt/Temperature scaling) to ``calibrating'' neural networks so that the output scores correspond more closely to probabilities/confidences.
It would be desirable to have this approach implemented in the next release of Ketos!https://gitlab.meridian.cs.dal.ca/public_projects/ketos/-/issues/112Option to specify relative sample occurrence in batch generator2021-03-30T12:36:20ZOliver KirsebomOption to specify relative sample occurrence in batch generatorIn an active-learning scenario where a human validates model outputs and identifies certain samples as particularly important, it would be useful to have a mechanism for over-sampling individual samples in the training process. One way t...In an active-learning scenario where a human validates model outputs and identifies certain samples as particularly important, it would be useful to have a mechanism for over-sampling individual samples in the training process. One way to achieve this could be to assign to each sample an integer to indicate how many times that sample should appear in a training epoch. By default, the value would be 1, i.e., each sample appears once in an epoch. This integer (which we could call `multiplicity') could be saved to a column in the hdf5 table. Then we could then point the BatchGenerator to this column in the hdf5 table using an argument like `mult_field` or something like that.
@fsfrazao , your thoughts?https://gitlab.meridian.cs.dal.ca/public_projects/ketos/-/issues/114transforms not available though Waveform or Spectrogram classes2021-04-13T17:50:50ZFabio Frazaotransforms not available though Waveform or Spectrogram classesSome transforms are not available through the Waveform or Spectrogram classes.
These are the ones I identified:
* apply_preemphasis
- https://docs.meridian.cs.dal.ca/ketos/modules/audio/utils/filter.html#ketos.audio.utils.filter.apply...Some transforms are not available through the Waveform or Spectrogram classes.
These are the ones I identified:
* apply_preemphasis
- https://docs.meridian.cs.dal.ca/ketos/modules/audio/utils/filter.html#ketos.audio.utils.filter.apply_preemphasis
* apply_median_filter
- https://docs.meridian.cs.dal.ca/ketos/modules/audio/utils/filter.html#ketos.audio.utils.filter.apply_median_filter
* filter_isolated_spots
- https://docs.meridian.cs.dal.ca/ketos/modules/audio/utils/filter.html#ketos.audio.utils.filter.filter_isolated_spots
Should they be included as methods in the appropriate classes, so they can use the same transform logs as other methods?Oliver KirsebomOliver Kirsebomhttps://gitlab.meridian.cs.dal.ca/public_projects/ketos/-/issues/118Behaviour of freeze_block and unfreeze_block2021-05-25T18:35:50ZOliver KirsebomBehaviour of freeze_block and unfreeze_blockI was using the ResNet freeze_block method as follows
```python
resnet.model.freeze_block([0,1,2,3,4,5]) #freeze all 6 blocks
# now train the model for a while ...
resnet.model.freeze_block([0,1,2,3,4,5]) #freeze only the first 5 blocks
...I was using the ResNet freeze_block method as follows
```python
resnet.model.freeze_block([0,1,2,3,4,5]) #freeze all 6 blocks
# now train the model for a while ...
resnet.model.freeze_block([0,1,2,3,4,5]) #freeze only the first 5 blocks
# continue training ...
```
However, eventually I realized that this was the wrong use of this method.
To achieve the desired result, I had to do as follows,
```python
resnet.model.freeze_block([0,1,2,3,4,5]) #freeze all 6 blocks
# now train the model for a while ...
resnet.model.unfreeze_block([5]) #unfreeze the last block
# continue training ...
```
I am wondering if other users might be fooled by this too?
Would it be more intuitive to implement the `freeze_block` in a manner so that past history of freezing layers is forgotten?
Or perhaps we could add an argument to enable such use?
Or at least highlight this in the doc string as a potential pitfall, e.g., via an example.
@fsfrazao what do you think?https://gitlab.meridian.cs.dal.ca/public_projects/ketos/-/issues/133Possible problem with implementation of group_detections2022-08-12T17:20:00ZOliver KirsebomPossible problem with implementation of group_detectionsIn the current implementation of the [group_detections](https://gitlab.meridian.cs.dal.ca/public_projects/ketos/-/blob/master/ketos/neural_networks/dev_utils/detection.py#L120), the duration of a grouped detection event can be shorter th...In the current implementation of the [group_detections](https://gitlab.meridian.cs.dal.ca/public_projects/ketos/-/blob/master/ketos/neural_networks/dev_utils/detection.py#L120), the duration of a grouped detection event can be shorter than the length of a single spectrogram sample if the samples are overlapping in time. For example, if a spectrogram with start=5.0s and duration=3.0s triggers a detection, but the subsequent spectrogram starting at start=5.5s does not trigger a detection, the grouped detection event is assumed to begin at start=5.0s and have a duration of only 0.5s. This is reflected in this unit test:
https://gitlab.meridian.cs.dal.ca/public_projects/ketos/-/blob/master/ketos/tests/neural_networks/test_detection.py#L198
However, as far as I can tell this constraining effect is only implemented for later samples, not earlier ones. In the above unit test, for example, the spectrogram sample starting at 4.5s and ending at 7.5s does not trigger a detection, yet the grouped detection event returned by the group_detection function still starts at 5.0s. Isn't this an inconsistent treatment of future and past, so to speak?
Infact, if we were to treat earlier negatives on par with later ones, the grouped detection event in the unit test would be have to be discarded altogether.
@fsfrazao , your thoughts?https://gitlab.meridian.cs.dal.ca/public_projects/ketos/-/issues/169Remove dropout from ResNet? (And other ketos architectures)2022-04-02T18:47:04ZOliver KirsebomRemove dropout from ResNet? (And other ketos architectures)In my own experience, training the ketos ResNet architecture with dropout > 0 results in (far) worse performance than dropout = 0. This appears to be consistent with observations made by others. See for example: https://www.kdnuggets.com...In my own experience, training the ketos ResNet architecture with dropout > 0 results in (far) worse performance than dropout = 0. This appears to be consistent with observations made by others. See for example: https://www.kdnuggets.com/2018/09/dropout-convolutional-networks.html . I'm wondering if we should simply remove the dropout argument from the definition of the ResNetBlock and ResNetArch in ketos: https://gitlab.meridian.cs.dal.ca/public_projects/ketos/-/blob/master/ketos/neural_networks/resnet.py#L100 ?
@bpadovese , @fsfrazao , your thoughts/experiences?
I'm not sure if the dropout has similar negative impact on the other ketos architectures (cnn, densenet, inception) ...https://gitlab.meridian.cs.dal.ca/public_projects/ketos/-/issues/188New format for ketos models2022-09-29T17:57:47ZFabio FrazaoNew format for ketos modelsThe current format for saved ketos models is an archive containing information on:
-pre-processing audio files: audio representation config
-the model: recipe + weights
I think it would be helpful to expand it to include more instructi...The current format for saved ketos models is an archive containing information on:
-pre-processing audio files: audio representation config
-the model: recipe + weights
I think it would be helpful to expand it to include more instructions on pre-processing (what to do with the data before it reaches the network) and post-processing (what to do with the network outputs.
This added information combined with corresponding functions to read and apply these instructions would allow for more general versions of some scripts in the ketos-scripts repo.
I would add:
-pre-processing audio files: audio representation + custom audio representation code + transform function + custom transform code
-the model: recipe + weights + custom architecture code
-post-processing: post-processing recipe + custom post-processing code
In summary, each option could use configuration files (such as the audio representation and model recipe already in use) and optionally code that allows operations not covered by the configuration files (i,e,: things that are not available as a ketos function/class)