KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#developer-tools #codex #typescript #ai-agents #claude-code #dsh #dsh-plugin #deepseek-harness #cli #macos #ai-agent #open-source

MorphQ-Quantum-Qiskit-Testing-ICSE-23

★ 10 Open GitHub ↗

Companion website of "MorphQ: Metamorphic Testing of the Qiskit Quantum Computing Platform", accepted at ICSE 23

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# MorphQ Software
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.7575881.svg)](https://doi.org/10.5281/zenodo.7575881)
[![DOI](https://img.shields.io/badge/doi.org/10.6084/m9.figshare.20703091.v2-blue.svg?style=flat&labelColor=whitesmoke&logo=data%3Aimage%2Fpng%3Bbase64%2CiVBORw0KGgoAAAANSUhEUgAAAB8AAAAfCAYAAAAfrhY5AAAJsklEQVR42qWXd1DTaRrHf%2BiB2Hdt5zhrAUKz4IKEYu9IGiGFFJJQ0gkJCAKiWFDWBRdFhCQUF3UVdeVcRQEBxUI3yY9iEnQHb3bdW1fPubnyz%2F11M7lvEHfOQee2ZOYzPyDv%2B3yf9%2Fk95YX4fx%2BltfUt08GcFEuPR4U9hDDZ%2FVngIlhb%2FSiI6InkTgLzgDcgfvtnovhH4BzoVlrbwr55QnhCtBW4QHXnFrZbPBaQoBh4%2FSYH2EnpBEtqcDMVzB93wA%2F8AFwa23XFGcc8CkT3mxz%2BfXWtq9T9IQlLIXYEuHojudb%2BCM7Hgdq8ydi%2FAHiBXyY%2BLjwFlAEnS6Jnar%2FvnQVhvdzasad0eKvWZKe8hvDB2ofLZ%2FZEcWsh%2BhyIuyO5Bxs2iZIE4nRv7NWAb0EO8AC%2FWPxjYAWuOEX2MSXZVgPxzmRL3xKz3ScGpx6p6QnOx4mDIFqO0w6Q4fEhO5IzwxlSwyD2FYHzwAW%2BAZ4fEsf74gCumykwNHskLM7taQxLYjjIyy8MUtraGhTWdkfhkFJqtvuVl%2F9l2ZquDfEyrH8B0W06nnpH3JtIyRGpH1iJ6SfxDIHjRXHJmdQjLpfHeN54gnfFx4W9QRnovx%2FN20aXZeTD2J84hn3%2BqoF2Tqr14VqTPUCIcP%2B5%2Fly4qC%2BUL3sYxSvNj1NwsVYPsWdMUfomsdkYm3Tj0nbV0N1wRKwFe1MgKACDIBdMAhPE%2FwicwNWxll8Ag40w%2BFfhibJkGHmutjYeQ8gVlaN%2BjO51nDysa9TwNUFMqaGbKdRJZFfOJSp6mkRKsv0rRIpEVWjAvyFkxNOEpwvcAVPfEe%2Bl8ojeNTx3nXLBcWRrYGxSRjDEk0VlpxYrbe1ZmaQ5xuT0u3r%2B2qe5j0J5uytiZPGsRL2Jm32AldpxPUNJ3jmmsN4x62z1cXrbedXBQf2yvIFCeZrtyicZZG2U2nrrBJzYorI2EXLrvTfCSB43s41PKEvbZDEfQby6L4JTj%2FfIwam%2B4%2BwucBu%2BDgNK05Nle1rSt9HvR%2FKPC4U6LTfvUIaip1mjIa8fPzykii23h2eanT57zQ7fsyYH5QjywwlooAUcAdOh5QumgTHx6aAO7%2FL52eaQNEShrxfhL6albEDmfhGflrsT4tps8gTHNOJbeDeBlt0WJWDHSgxs6cW6lQqyg1FpD5ZVDfhn1HYFF1y4Eiaqa18pQf3zzYMBhcanlBjYfgWNayAf%2FASOgklu8bmgD7hADrk4cRlOL7NSOewEcbqSmaivT33QuFdHXj5sdvjlN5yMDrAECmdgDWG2L8P%2BAKLs9ZLZ7dJda%2BB4Xl84t7QvnKfvpXJv9obz2KgK8dXyqISyV0sXGZ0U47hOA%2FAiigbEMECJxC9aoKp86re5O5prxOlHkcksutSQJzxZRlPZmrOKhsQBF5zEZKybUC0vVjG8PqOnhOq46qyDTDnj5gZBriWCk4DvXrudQnXQmnXblebhAC2cCB6zIbM4PYgGl0elPSgIf3iFEA21aLdHYLHUQuVkpgi02SxFdrG862Y8ymYGMvXDzUmiX8DS5vKZyZlGmsSgQqfLub5RyLNS4zfDiZc9Edzh%2FtCE%2BX8j9k%2FqWB071rcZyMImne1SLkL4GRw4UPHMV3jjwEYpPG5uW5fAEot0aTSJnsGAwHJi2nvF1Y5OIqWziVCQd5NT7t6Q8guOSpgS%2Fa1dSRn8JGGaCD3BPXDyQRG4Bqhu8XrgAp0yy8DMSvvyVXDgJcJTcr1wQ2BvFKf65jqhvmxXUuDpGBlRvV36XvGjQzLi8KAKT2lYOnmxQPGorURSV0NhyTIuIyqOmKTMhQ%2BieEsgOgpc4KBbfDM4B3SIgFljvfHF6cef7qpyLBXAiQcXvg5l3Iunp%2FWv4dH6qFziO%2BL9PbrimQ9RY6MQphEfGUpOmma7KkGzuS8sPUFnCtIYcKCaI9EXo4HlQLgGrBjbiK5EqMj2AKWt9QWcIFMtnVvQVDQV9lXJJqdPVtUQpbh6gCI2Ov1nvZts7yYdsnvRgxiWFOtNJcOMVLn1vgptVi6qrNiFOfEjHCDB3J%2BHDLqUB77YgQGwX%2Fb1eYna3hGKdlqJKIyiE4nSbV8VFgxmxR4b5mVkkeUhMgs5YTi4ja2XZ009xJRHdkfwMi%2BfocaancuO7h%2FMlcLOa0V%2FSw6Dq47CumRQAKhgbOP8t%2BMTjuxjJGhXCY6XpmDDFqWlVYbQ1aDJ5Cptdw4oLbf3Ck%2BdWkVP0LpH7s9XLPXI%2FQX8ws%2Bj2In63IcRvOOo%2BTTjiN%2BlssfRsanW%2B3REVKoavBOAPTXABW4AL7e4NygHdpAKBscmlDh9Jysp4wxbnUNna3L3xBvyE1jyrGIkUHaqQMuxhHElV6oj1picvgL1QEuS5PyZTEaivqh5vUCKJqOuIgPFGESns8kyFk7%2FDxyima3cYxi%2FYOQCj%2F%2B9Ms2Ll%2Bhn4FmKnl7JkGXQGDKDAz9rUGL1TIlBpuJr9Be2JjK6qPzyDg495UxXYF7JY1qKimw9jWjF0iV6DRIqE%2B%2FeWG0J2ofmZTk0mLYVd4GLiFCOoKR0Cg727tWq981InYynvCuKW43aXgEjofVbxIqrm0VL76zlH3gQzWP3R3Bv9oXxclrlO7VVtgBRpSP4hMFWJ8BrUSBCJXC07l40X4jWuvtc42ofNCxtlX2JH6bdeojXgTh5TxOBKEyY5wvBE%2BACh8BtOPNPkApjoxi5h%2B%2FFMQQNpWvZaMH7MKFu5Ax8HoCQdmGkJrtnOiLHwD3uS5y8%2F2xTSDrE%2F4PT1yqtt6vGe8ldMBVMEPd6KwqiYECHDlfbvzphcWP%2BJiZuL5swoWQYlS%2Br7Yu5mNUiGD2retxBi9fl6RDGn4Ti9B1oyYy%2BMP5G87D%2FCpRlvdnuy0PY6RC8BzTA40NXqckQ9TaOUDywkYsudxJzPgyDoAWn%2BB6nEFbaVxxC6UXjJiuDkW9TWq7uRBOJocky9iMfUhGpv%2FdQuVVIuGjYqACbXf8aa%2BPeYNIHZsM7l4s5gAQuUAzRUoT51hnH3EWofXf2vkD5HJJ33vwE%2FaEWp36GHr6GpMaH4AAPuqM5eabH%2FhfG9zcCz4nN6cPinuAw6IHwtvyB%2FdO1toZciBaPh25U0ducR2PI3Zl7mokyLWKkSnEDOg1x5fCsJE9EKhH7HwFNhWMGMS7%2BqxyYsbHHRUDUH4I%2FAheQY7wujJNnFUH4KdCju83riuQeHU9WEqNzjsJFuF%2FdTDAZ%2FK7%2F1WaAU%2BAWymT59pVMT4g2AxcwNa0XEBDdBDpAPvgDIH73R25teeuAF5ime2Ul0OUIiG4GpSAEJeYW9wDTf43wfwHgHLKJoPznkwAAAABJRU5ErkJggg%3D%3D)](http://doi.org/10.6084/m9.figshare.20703091.v2)
[<img src="https://img.shields.io/badge/dockerhub-mattepalte/morphq-important.svg?logo=LOGO">](https://hub.docker.com/r/mattepalte/morphq)

MorphQ is a software to automatically test the Qiskit quantum computing platform using metamorphic testing.
The MorphQ software is open source and you find all the relevant source files in the [lib](lib) folder.

You can run MorphQ with two objectives:
1. [Replication Package Level 1](#reproduce-the-paper-figures-level-1): reproduce paper's results.
2. [Replication Package Level 2](#run-morphq-for-a-new-testing-campaign-level-2): run new testing campaigns to find new bugs in Qiskit. or compare your new testing method against MorphQ. (Note: you can also use our method in the Qdiff configuration to compare against QDiff transformations.)

# Paper

[MorphQ: Metamorphic Testing of the Qiskit Quantum Computing Platform](https://software-lab.org/publications/icse2023_MorphQ.pdf)

Matteo Paltenghi, Michael Pradel.
IEEE/ACM International Conference on Software Engineering

# Getting Started

1. Check that your setup meets the [REQUIREMENTS.md](REQUIREMENTS.md).
2. Follow the installation instructions in [INSTALL.md](INSTALL.md).

Note: if you are more familiar with docker or you are having trouble with the installation on your system, you can also build a docker image based on Ubuntu.
We provide a [Dockerfile](Dockerfile) which will configure an Ubuntu container containing all you need to run MorphQ. See [DOCKER.md](DOCKER.md) for instructions. We recommend reading this README first and then the docker instructions.

# Reproduce the Paper Figures (Level 1)

This replication level allows other researcher to independently reproduce the results of our paper starting from the experimental data we collected during our empirical evaluation.

Follow this steps:
1. Download the datasets used in our evaluation section from this [link](https://doi.org/10.6084/m9.figshare.20703091.v2).

2. unzip the archive. And copy the folders `qmt_v52` and `qmt_v53` in the `data` directory.

3. to open jupyter notebook, run the following in your terminal:
    ```bash
    conda activate MorphQ
    jupyter notebook
    ```

4. In the jupyter notebook web interface, navigate to and execute top-to-bottom the following notebook: [notebooks/RQs_Reproduce_Analysis_Results.ipynb](notebooks/RQs_Reproduce_Analysis_Results.ipynb).

**Inspect the warnings found during our empirical evaluation:**
To see the warnings described in our empirical evaluation of the paper refer to the [`warnings/program_pairs`](warnings/program_pairs) folder.
There you will find a subfolder for each program pair which contains:
- the source quantum program: `source.py`
- the follow-up quantum program generated via metamorphic transformations: `follow-up.py`
- the metadata of the generation of the program pair: `metadata.json`
- the metadata of the execution of the program pair: `metadata_exec.json`
- an anonymized screenshot of the issue page: `program_code.pdf`

# Run MorphQ For a New Testing Campaign (Level 2)

This guide is intended for researchers who want to:
1. run their own testing campaigns to find new bugs in Qiskit with MorphQ,
2. compare MorphQ with a novel automatic testing approach of the Qiskit platform.

A MorphQ run is configured via a YAML file, read next to learn how to configure your own run. The one used for our experiment are store in the [config](config) folder:
- [qmt_v53.yaml](config/qmt_v53.yaml): configuration file used for our experiment for the actual **MorphQ**.
- [qmt_v52.yaml](config/qmt_v52.yaml): configuration file used for our experiment for the **QDiff-Transformations**.

Some important fields in the config file are:
- `experiment_folder`: the folder where the data will be stored.
- `generation_strategy`: the strategy to use to generate the source quantum programs. You can tweak the parameters to define their size: `min_n_qubits`, `max_n_qubits`, `min_n_ops`, `max_n_ops`; and the basic gates used: `gate_set`. Note: in case you want to change the program generator at all, you can simply plug a different object in the `generator_object` field, remember to use the same interface of the `QiskitFuzzer` class in the [lib/generation_strategy_python.py](/lib/generation_strategy_python.py) file.
- `max_transformations_per_program`: the max number of metamorphic transformations to apply to each program in a chain.
- `transformation_mode`: the metamorphic transformation to apply, either `morphq` or `qdiff`.
- `morphq_metamorphic_strategies`: the metamorphic transformation to apply for the `morphq` mode.
- `qdiff_metamorphic_strategies`: the metamorphic transformation to apply for the `qdiff` mode.
- `coverage_settings_filepath`: the  path to the coverage settings, typically a `.cover` file. You can find some examples in the [config](config) folder.

We prepared a convenient way to generate a new configuration file from a template.

1. run the following command:
    ```bash
    python3 -m lib.generate_new_config --version 01
    ```
2. type the number to select the `morphq_demo.yaml` template. You should get the following output:
    ```python3 -m lib.generate_new_config --version 01
    Available Templates:
    1. morphq_demo.yaml
    2. morphq.yaml
    3. qdiff_demo.yaml
    4. qdiff.yaml
    Choose a template: 1
    Setting file created: ./config/qmt_v01.yaml

Creating coverage file:
    Coverage file created:: config/qmt_v01.cover

Creating experiment folder:
    mkdir -p data/qmt_v01/
    Experiment folder created: data/qmt_v01/

Excluding experiment folder from gitignore:
    echo 'data/qmt_v01/' >> .gitignore
    Added to .gitignore: data/qmt_v01/

Example commands:

- Simple experiment run (no monitoring):
    python3 -m lib.qmt config/qmt_v01.yaml

- Experiment run (monitoring via screen):
    screen -L -Logfile data/qmt_v01/log_fuzzy.txt -S qmt_v01 python3 -m lib.qmt config/qmt_v01.yaml
    ```
3. The you can run the MorphQ, either without the monitoring or with the monitoring (using screen), as mentioned in the output.
    ```bash
    - Simple experiment run (no monitoring):
    python3 -m lib.qmt config/qmt_v01.yaml

- Experiment run (monitoring via screen):
    screen -L -Logfile data/qmt_v01/log_fuzzy.txt -S qmt_v01 python3 -m lib.qmt config/qmt_v01.yaml
    ```
    Note: if you do not have screen installed, you can install it with `sudo apt install screen`.
4. If you used the monitoring setup, you will find the log at the specified path (e.g., `data/qmt_v01/log_fuzzy.txt`). And it should contain what you see in the terminal. An example is shown belowe:
    ```bash
    tarting loop... [no timer]
    Data file:  data/qmt_v01/coverage.db
    --------- New programs pair... [timer: 120 sec] (2023-01-17-13:25:56) ----------
    Applying: MetamorphicTransformation(ToQasmAndBack)
    Follow: add 'qc' conversion to and from QASM (before: EXECUTION)
    Applying: MetamorphicTransformation(ChangeTargetBasis)
    Follow: gateset replaced with:  ['ccx', 'h']
    Applying: MetamorphicTransformation(ChangeTargetBasis)
    Follow: gateset replaced with:  ['rx', 'ry', 'rz', 'p', 'cx']
    N. applied transformations:  3
    Executing: 381b90a6f912482f90076c971f2f7074 (2023-01-17-13:25:57)
    Exceptions from execution: {'source': None, 'followup': '"Cannot unroll the circuit to the given basis, [\'rx\', \'ry\', \'rz\', \'p\', \'cx\']. Instruction id not found in equivalence library and no rule found to expand."'}
    --------- New programs pair... [timer: 120 sec] (2023-01-17-13:25:57) ----------
    Applying: MetamorphicTransformation(ToQasmAndBack)
    Follow: add 'qc' conversion to and from QASM (before: OPTIMIZATION_LEVEL)
    Applying: MetamorphicTransformation(InjectParameters)
    Follow: from concrete values to parameters (12/18):{'p_a0b579': 0.14709460241864125, 'p_2d234e': 2.7426492410962213, 'p_4aca1b': 1.4655643254447606, 'p_45c5e7': 1.2941054775314977, 'p_079834': 1.297948790230138, 'p_7cd8ae': 4.883628121581527, 'p_afbea1': 5.665169039628415, 'p_761208': 1.8300494450884037, 'p_6baf69': 2.5947845640969898, 'p_098317': 4.818497722767661, 'p_134a27': 5.236334390147421, 'p_e21ca9': 3.219605809291016}
    Applying: MetamorphicTransformation(AddUnusedRegister)
    Follow: add QuantumRegister(1)
    N. applied transformations:  3
    Executing: f41d90fd3cb94e6f925f07c0a20e2a48 (2023-01-17-13:25:58)
    Exceptions from execution: {'source': None, 'followup': None}
    i*: 0
    --------- New programs pair... [timer: 120 sec] (2023-01-17-13:25:59) ----------
    Applying: MetamorphicTransformation(ChangeOptLevel)
    Follow: optimization level changed: 1 -> 0
    Applying: MetamorphicTransformation(ToQasmAndBack)
    Follow: add 'qc' conversion to and from QASM (before: EXECUTION)
    Applying: MetamorphicTransformation(ChangeOptLevel)
    Follow: optimization level changed: 0 -> 2
    Applying: MetamorphicTransformation(ChangeCouplingMap)
    Follow: coupling map changed: None -> [[0, 1], [1, 0], [1, 2], [2, 1]]
    N. applied transformations:  4
    Executing: 7e5582035ba3412eb8d8dba421cf6549 (2023-01-17-13:25:59)
    Exceptions from execution: {'source': None, 'followup': None}
    i*: 0
    ```
5. Congratulation! MorphQ is running!
6. If you want to stop the experiment, you can do it by pressing `Ctrl+C` in the terminal. If you used the monitoring setup, you can also do it by typing `screen -S qmt_v01 -X quit` in the terminal.

## Extra Utilities: Inspect Warnings of Your Run

Here we describe where to find and how to inspect the warnings generated by `MorphQ`.

The relevant warnings of your run will be stored in a the `qfl.db` sqlite database in the `data/qmt_vXX/` folder.
To inspect them use the notebook [`notebooks/Demo_Warnings_Manual_Inspection.ipynb`](notebooks/Demo_Warnings_Manual_Inspection.ipynb).
Remember to update the `EXP_FOLDER = "qmt_v53"` cell to point ot the actual `qmt_vXX` folder of your run.

## Extra Run: QDiff-Transformations

To run the QDiff pipeline, just pick a `qdiff` template when prompted by running:
```bash
python3 -m lib.generate_new_config --version 02
```

## Extra Details: Using Your Own Settings
Before starting your test run, you need to prepare some configuration files in the data [`config`](config) folder.

1. create a `qmt_vXX.yaml` file in the `config` folder. On this file you can set almost all the parameters of the test. Including the metamorphic relationships to use (via the `metamorphic_strategies` field) and the program generation mechanism (via the `generation_strategy` field).

**IMPORTANT**: Remember to update the two field: `experiment_folder` to point to the data folder (e.g., `data/qmt_vXX/`) and `coverage_settings_filepath` to point to the file containing the coverage settings (e.g., `config/qmt_vXX.cover`).

2. create a `qmt_vXX.cover` file in the `config` folder. On this file you can set the coverage settings.

**IMPORTANT**: update the `data_file` field to specify the final name of the coverage database in the correct data folder (e.g., `data/qmt_vXX/.coverage`).

3. (optional) to avoid that the generated programs are processed by the git installation add the data folder to your `.gitignore` file.
```
data/qmt_vXX/*
```

# Troubleshooting
Some questions you might ask yourself are:
1. Did you install and activate the conda environment? via `conda activate MorphQ`

MorphQ-Quantum-Qiskit-Testing-ICSE-23

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README