AuditionMaster App

Introduction

🎵🎻 Excited to share a unique project that fuses my passions for music and technology!

As a former professional symphony orchestra musician, I’ve faced the rigors of preparing for auditions firsthand. Inspired by this, I’ve created a tool to aid fellow musicians in honing their technical skills for orchestral auditions.

The app I’ve built allows musicians to upload or record a performance excerpt. It then analyzes the recording, providing crucial performance metrics such as tempo, volume, clarity, and tuning, displayed through numerical scores and time-series graphs. Users can even watch their musical progress unfold on these metric plots as the recording plays in the app.

While the app offers valuable insights, it’s important to note that the interpretation of the results rests with the musician. The app provides an objective analysis but doesn’t discern artistic intention. It informs you if a part was loud, out of tempo, or out of tune, but you decide how these insights apply to your unique style and interpretive choices. This makes the tool a personal journey of improvement, with scores primarily intended for self-comparison over time.

Built on Flask and powered by the Librosa library for music and audio analysis, the UI was designed using HTML, JavaScript, CSS, and ChatGPT, ensuring an intuitive and user-friendly experience.

I believe this app can transform how musicians prepare for auditions. Check out the attached demo video.Here’s to empowering musicians with tech! 🎶🎼🎹

Here’s a demo of the app:

The main file: app.py

from flask import Flask, render_template, request
from werkzeug.utils import secure_filename
import os
import librosa
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
import mir_eval


app = Flask(__name__)

UPLOAD_FOLDER = 'static/uploads'

ALLOWED_EXTENSIONS = {'mp3'}

app.config['UPLOAD_FOLDER'] = UPLOAD_FOLDER

@app.route('/')
def home():
    return render_template('index.html')

@app.route('/upload', methods=['POST'])
def upload_file():
    if 'audiofile' in request.files:
        audiofile = request.files['audiofile']
        if audiofile.filename == '':
            return 'No file selected'
        if audiofile and allowed_file(audiofile.filename):
            filename = secure_filename(audiofile.filename)
            audiofile.save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
            return process_audio(filename)
        else:
            return 'Invalid file format'
    else:
        return 'No file part'

def allowed_file(filename):
    return '.' in filename and filename.rsplit('.', 1)[1].lower() in ALLOWED_EXTENSIONS


def process_audio(filename):
    # Load the audio file
    file_path = os.path.join(app.config['UPLOAD_FOLDER'], filename)
    y, sr = librosa.load(file_path)

    # Prepare time array for plots
    times_array = librosa.frames_to_time(np.arange(len(y)), sr=sr)

    # pYIN pitch tracking
    f0, voiced_flag, voiced_probs = librosa.pyin(y, fmin=librosa.note_to_hz('C2'), fmax=librosa.note_to_hz('C7'))

    # times corresponding to f0 estimates
    times = librosa.times_like(f0)

    # Beat tracking
    tempo, beats = librosa.beat.beat_track(y=y, sr=sr)

    # Beat frames to time
    beat_times = librosa.frames_to_time(beats, sr=sr)

    # Calculate the time difference between beats
    beat_diff = np.diff(beat_times)

    # Convert f0 values into musical notes
    notes = mir_eval.melody.hz2cents(f0, base_frequency=442.0) / 100

    # Remove unvoiced parts
    notes = notes[voiced_flag]

    # Filter times with voiced_flag
    times_voiced = times[voiced_flag]

    # Compute the short-time Fourier transform (STFT)
    D = librosa.stft(y)

    # Separate the harmonic and percussive components
    H, P = librosa.decompose.hpss(D)

    # Convert the magnitude spectrogram to dB-scaled spectrogram
    H_db = librosa.amplitude_to_db(np.abs(H), ref=np.max)

    # Compute MFCCs
    mfccs = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=13)

    mfcc_std = np.std(mfccs, axis=1)

    # Compute "timbre score" 
    timbre_score = 1 / np.mean(mfcc_std)

    # Create subplots using gridspec
    fig = plt.figure(figsize=(14, 16))
    gs = gridspec.GridSpec(10, 1)

    # Waveform plot
    ax1 = fig.add_subplot(gs[0, 0])
    ax1.plot(times_array, y)
    ax1.set_title('Waveform')

    # RMS energy over time plot
    ax2 = fig.add_subplot(gs[1, 0])
    rms = librosa.feature.rms(y=y)[0]
    rms_times = librosa.frames_to_time(np.arange(len(rms)), sr=sr)
    ax2.plot(rms_times, rms)
    ax2.set_title('Loudness (RMS Energy) Over Time')

    # Inter-beat intervals over time plot
    ax3 = fig.add_subplot(gs[2, 0])
    ax3.plot(beat_times[:-1], beat_diff)
    ax3.set_title('Tempo (Inter-beat Intervals) Over Time')
    ax3.set_xlabel('Time (s)')
    ax3.set_ylabel('Interval (s)')

    # Pitch track plot
    ax4 = fig.add_subplot(gs[3, 0])
    ax4.plot(times_voiced, notes, label='f0')
    ax4.legend(loc='upper right')
    ax4.set_xlabel('Time')
    ax4.set_ylabel('Pitch')
    ax4.set_title('Pitch track')
    ax4.grid(True)

    # Harmonic spectrogram plot
    ax5 = fig.add_subplot(gs[4:7, 0])
    librosa.display.specshow(H_db, sr=sr, x_axis='time', y_axis='log', ax=ax5)
    ax5.set_title('Harmonic Spectrogram')

    # MFCCs plot
    ax6 = fig.add_subplot(gs[7:10, 0])
    librosa.display.specshow(mfccs, sr=sr, x_axis='time', ax=ax6)
    ax6.set_title('MFCCs')

    # Save the combined image
    plots_folder = 'static/plots'
    os.makedirs(plots_folder, exist_ok=True)
    combined_path = f'{plots_folder}/combined_plots.png'
    plt.tight_layout(pad=0)  # tight_layout with padding set to 0
    plt.savefig(combined_path, bbox_inches='tight', pad_inches=0)  # saving figure with additional arguments
    #plt.close(fig)

    rms = librosa.feature.rms(y=y)[0]
    loudness_score = np.std(rms)
    beat_diff_score = 1 / np.std(beat_diff)
    pitch_score = 1 / np.std(notes)
    contrast = librosa.feature.spectral_contrast(S=np.abs(H), sr=sr)
    contrast_score = np.mean(contrast)

    total_score = (loudness_score + beat_diff_score + pitch_score + contrast_score + timbre_score) / 5

    scores = {
        'loudness_score': loudness_score,
        'beat_diff_score': beat_diff_score,
        'pitch_score': pitch_score,
        'contrast_score': contrast_score,
        'timbre_score': timbre_score
    }

    plots = {
        'combined_plots': combined_path
    }

    return render_template('results.html', scores=scores, plots=plots, total_score=total_score, filename=filename)


if __name__ == '__main__':
    app.run(debug=True)

index.html file

<!DOCTYPE html>
<html>
<head>
    <title>Orchestra Auditions Metrics</title>
    <link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='style.css') }}">
    <style>
        body {
            background-color: #f2f2f2;
            font-family: Arial, sans-serif;
            text-align: center;
            padding: 20px;
        }

        h1 {
            color: #333333;
            font-size: 36px;
            margin-bottom: 30px;
        }

        .container {
            display: flex;
            justify-content: space-around;
        }

        .section {
            margin-bottom: 50px;
            flex-basis: 45%;
        }

        .section-title {
            color: #333333;
            font-size: 24px;
            margin-bottom: 10px;
        }

        .section-description {
            color: #666666;
            font-size: 16px;
            margin-bottom: 20px;
        }

        .upload-form,
        .record-form {
            display: flex;
            flex-direction: column;
            align-items: center;
            margin-top: 30px;
        }

        .upload-form input[type="file"],
        .record-form button {
            padding: 10px 20px;
            font-size: 16px;
            border: none;
            background-color: #007bff;
            color: #ffffff;
            box-shadow: 0 2px 6px rgba(0, 0, 0, 0.1);
            margin-bottom: 20px;
            cursor: pointer;
        }

        .upload-form input[type="file"]:hover,
        .upload-form input[type="file"]:focus,
        .record-form button:hover,
        .record-form button:focus {
            background-color: #0056b3;
        }
    </style>
    <script>
        // Add your JavaScript code here
    </script>
</head>
<body>
    <h1>Welcome to "Orchestra Auditions Metrics"</h1>

    <div class="container">
        <div class="section">
            <h2 class="section-title">Upload High-Quality Recording</h2>
            <p class="section-description">Enhance your orchestra auditions with advanced metrics and analysis.</p>
            <form class="upload-form" action="{{ url_for('upload_file') }}" method="post" enctype="multipart/form-data">
                <input id="audio-file-input" type="file" name="audiofile" accept="audio/*" onchange="handleFileInputChange()">
                <button type="submit">Upload</button>
            </form>
        </div>

        <div class="section">
            <h2 class="section-title">Record Inside the App</h2>
            <p class="section-description">Capture your performance directly and get instant feedback.</p>
            <form class="record-form" onsubmit="event.preventDefault()">
                <audio id="audio-player" controls></audio>
                <button class="record-btn" type="button" onclick="startRecording()">Record</button>
                <button class="record-btn" type="button" onclick="stopRecording()">Stop Recording</button>
                <button type="submit">Upload</button>
            </form>
        </div>
    </div>
</body>
</html>

results.html file

<!DOCTYPE html>
<html>
<head>
    <title>Results</title>
    <link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='style.css') }}">
    <style>
        body {
            background-color: #f2f2f2;
            font-family: Arial, sans-serif;
            text-align: center;
            padding: 20px;
        }

        h1 {
            color: #333333;
            font-size: 36px;
            margin-bottom: 30px;
        }

        h2 {
            color: #333333;
            font-size: 24px;
            margin-bottom: 20px;
        }

        table {
            margin: 0 auto;
            border-collapse: collapse;
            width: 80%;
        }

        th, td {
            padding: 10px;
            text-align: center;
            font-size: 18px;
        }

        th {
            background-color: #007bff;
            color: #ffffff;
        }

        td {
            background-color: #ffffff;
        }

        .total-score {
            font-size: 36px;
            margin-top: 20px;
            font-weight: bold;
            color: #007bff;
        }

        .plots-container {
            display: flex;
            justify-content: center;
            margin-bottom: 30px;
        }

        .plot {
            position: relative;
        }

        .plot img {
            max-width: 100%;
        }

        .overlay {
            position: absolute;
            top: 0;
            left: 0;
            bottom: 0;
            background-color: rgba(0, 123, 255, 0.3);
            width: 0;
        }
    </style>
</head>
<body>
    <h1>Results</h1>

    <table>
        <tr>
            <th>Metric</th>
            <th>Score</th>
            <th>Explanation</th>
        </tr>
        <tr>
            <td>Dynamics</td>
            <td>{{ scores.loudness_score }}</td>
            <td>Measures the overall loudness of the performance</td>
        </tr>
        <tr>
            <td>Tempo</td>
            <td>{{ scores.beat_diff_score }}</td>
            <td>Assesses the accuracy and consistency of the timing and rhythm</td>
        </tr>
        <tr>
            <td>Intonation</td>
            <td>{{ scores.pitch_score }}</td>
            <td>Evaluates the accuracy and correctness of the pitch</td>
        </tr>
        <tr>
            <td>Tone Quality</td>
            <td>{{ scores.contrast_score }}</td>
            <td>Examines the richness, purity, and expressiveness of the tone</td>
        </tr>
        <tr>
            <td>Timbre</td>
            <td>{{ scores.timbre_score }}</td>
            <td>Assesses the unique sound characteristics and color of the performance</td>
        </tr>
    </table>

    <h2>Total Score</h2>
    <div class="total-score">{{ total_score }}</div>

    <audio controls id="audio-player">
        <source src="{{ url_for('static', filename='uploads/' + filename) }}" type="audio/mpeg">
        Your browser does not support the audio element.
    </audio>

    <div class="plots-container">
        <div class="plot">
            <img src="{{ plots.combined_plots }}" alt="Combined Plots">
            <div class="overlay"></div>
        </div>
    </div>

    <script src="https://unpkg.com/wavesurfer.js"></script>
    <script>
        var plot = document.querySelector(".plot");
        var overlay = plot.querySelector(".overlay");
        var audioPlayer = document.getElementById("audio-player");
        var plotImage = plot.querySelector("img");
      
        function syncPlot(time) {
          var progressRatio = time / audioPlayer.duration;
          var plotWidth = plotImage.clientWidth;
          var overlayWidth = progressRatio * plotWidth * 0.87;
          overlay.style.width = `${overlayWidth}px`;
        }
      
        audioPlayer.addEventListener("timeupdate", function () {
          syncPlot(this.currentTime);
        });
      
        audioPlayer.addEventListener("loadedmetadata", function () {
          syncPlot(0);
        });
      </script>
      
</body>
</html>

style.css file

body {
    margin: 0;
    padding: 0;
    box-sizing: border-box;
}

.plots-container {
    width: 100%;
    display: flex;
    justify-content: center;
    align-items: flex-start; /* Align items to the top */
}


.plot {
    position: relative;
    width: 100%; /* Adjust as necessary */
    margin: 1em;
    box-sizing: border-box; 
}

.plot img {
    width: auto;  /* Adjust width automatically to image's width */
    height: auto;
}


.overlay {
    position: absolute;
    top: 0;
    left: 0;
    bottom: 0;
    background-color: rgba(0, 123, 255, 0.3);
    width: 100%; /* Adjust the width as desired */
    transition: width 0.2s ease; /* Add transition effect for smooth resizing */
    margin-left: 77px;
  }

Conclusion

The full code is published on my GitHub.

Please feel free to contact me through LinkedIn with ideas for future iterations of this project.