# How to build on FX1000
## (1) Clone the repository
Our repository is private, so you need to register the public key at first.
1. Generate a public key
   ```
   $ ssh-keygen -t ed25519
   $ cat .ssh/id_ed25519.pub
   ```
1. Register the public key
   * [Settings] --> [SSH and GPG keys] --> push [New SSH key]
   * or [this page](https://github.com/settings/ssh/new)
1. clone the repository
   * You cannot use the original home directory, but can use `/work/ga28/{YourID}`.
   ```
   $ cd /work/ga28/a2800X/
   $ git clone git@github.com:Fujitsu-UTokyo-QDD/QDD.git
   ```

## (2) Build
1. Create `build.sh` file in QDD directory.
   ```
   $ cd QDD
   $ cat build.sh
   #!/bin/bash
   #PJM -L rscgrp=short-o
   #PJM -L node=1
   #PJM -L elapse=00:05:00
   #PJM -g ga28
   
   module load python/3.8.9
   CC="mpifcc -Kfast -Nclang -msve-vector-bits=512 -fno-builtin" CXX="mpiFCC -Kfast -Nclang -msve-vector-bits=512 -fno-builtin" Python_ROOT_DIR=/work/opt/local/aarch64/apps/fj/1.2.31/python/3.8.9 Boost_DIR=/work/ga28/share/boost_1_81_0-aarch64/lib/cmake/Boost-1.81.0 /work/ga28/share/cmake-3.26.4-linux-aarch64/bin/cmake -B build -DCMAKE_BUILD_TYPE=Release -DisMPI=ON
   /work/ga28/share/cmake-3.26.4-linux-aarch64/bin/cmake --build build -j
   ```
   * Points
      * Boost library is already built with gcc, and put in the shared directory. (Fujitsu compiler (mpifcc/mpiFCC) does not work well.)
      * Newer cmake is put in the shared directory.
      * QDD should be compiled with Fujitsu compiler (for profiling).
1. Submit the job
   ```
   $ pjsub build.sh
   ```
1. Check the job id & status
   ```
   $ pjstat
   ```
   * You can find the stdout/stderr from `build.sh.{JobID}` file.

## (3) Run
1. Create `run.sh` file in QDD directory.
   ```
   $ cat run.sh
   #!/bin/bash
   #PJM -L rscgrp=short-o
   #PJM -L node=4
   #PJM --mpi proc=1
   #PJM -L elapse=01:00:00
   #PJM -g ga28

   mpiexec -np 4 ./build/test/mpi_test_grover 30
   ```
   * `#PJM -L node=4` represents the number of nodes.
1. Submit the job
   ```
   $ pjsub run.sh
   ```
1. Check the job id & status
   ```
   $ pjstat
   ```
   * You can find the stdout/stderr from `run.sh.{JobID}` file.

## Tips
* You can use FX1000 interactively with the following command.
  ```
  $ pjsub --interact -g ga28 -L rscgrp=interactive-o,node=1
  ```
  * If the number of node is 1, the maximum execution time is 60mins.
  * If more, it is 10mins.
  * You cannot use multiple interactive jobs at the same time.