## 1. Project Overview & Quickstart (grosser/parallel) Parallel ============== [](https://rubygems.org/gems/parallel) [](https://github.com/grosser/parallel/actions/workflows/actions.yml) Run any code in parallel Processes(> use all CPUs), Threads(> speedup blocking operations), or Ractors(> use all CPUs). Best suited for map-reduce or e.g. parallel downloads/uploads. Install ======= ```Bash gem install parallel ``` Usage ===== ```Ruby # 2 CPUs -> work in 2 processes (a,b + c) results = Parallel.map(['a','b','c']) do |one_letter| SomeClass.expensive_calculation(one_letter) end # 3 Processes -> finished after 1 run results = Parallel.map(['a','b','c'], in_processes: 3) { |one_letter| SomeClass.expensive_calculation(one_letter) } # 3 Threads -> finished after 1 run results = Parallel.map(['a','b','c'], in_threads: 3) { |one_letter| SomeClass.expensive_calculation(one_letter) } # 3 Ractors -> finished after 1 run results = Parallel.map(['a','b','c'], in_ractors: 3, ractor: [SomeClass, :expensive_calculation]) ``` Same can be done with `each` ```Ruby Parallel.each(['a','b','c']) { |one_letter| ... } ``` or `each_with_index`, `map_with_index`, `flat_map` Produce one item at a time with `lambda` (anything that responds to `.call`) or `Queue`. ```Ruby items = [1,2,3] Parallel.each( -> { items.pop || Parallel::Stop }) { |number| ... } ``` Also supports `any?` or `all?` ```Ruby Parallel.any?([1,2,3,4,5,6,7]) { |number| number == 4 } # => true Parallel.all?([1,2,nil,4,5]) { |number| number != nil } # => false ``` Processes/Threads are workers, they grab the next piece of work when they finish. ### Processes - Speedup through multiple CPUs - Speedup for blocking operations - Variables are protected from change - Extra memory used - Child processes are killed when your main process is killed through Ctrl+c or kill -2 ### Threads - Speedup for blocking operations - Variables can be shared/modified - No extra memory used ### Ractors - Ruby 3.0+ only - Speedup for blocking operations - No extra memory used - Very fast to spawn - Experimental and unstable - `start` and `finish` hooks are called on main thread - Variables must be passed in `Parallel.map([1,2,3].map { |i| [i, ARGV, local_var] }, ...` - use `Ractor.make_shareable` to pass in global objects ### ActiveRecord #### Connection Lost - Multithreading needs connection pooling, forks need reconnects - Adjust connection pool size in `config/database.yml` when multithreading ```Ruby # reproducibly fixes things (spec/cases/map_with_ar.rb) Parallel.each(User.all, in_processes: 8) do |user| user.update_attribute(:some_attribute, some_value) end User.connection.reconnect! # maybe helps: explicitly use connection pool Parallel.each(User.all, in_threads: 8) do |user| ActiveRecord::Base.connection_pool.with_connection do user.update_attribute(:some_attribute, some_value) end end # maybe helps: reconnect once inside every fork Parallel.each(User.all, in_processes: 8) do |user| @reconnected ||= User.connection.reconnect! || true user.update_attribute(:some_attribute, some_value) end ``` #### NameError: uninitialized constant A race happens when ActiveRecord models are autoloaded inside parallel threads in environments that lazy-load, like development, test, or migrations. To fix, autoload classes before the parallel block with either `require ''` or `ModelName.class`. ### Break ```Ruby Parallel.map([1, 2, 3]) do |i| raise Parallel::Break # -> stops after all current items are finished end ``` ```Ruby Parallel.map([1, 2, 3]) { |i| raise Parallel::Break, i if i == 2 } == 2 ``` ### Kill Only use if whatever is executing in the sub-command is safe to kill at any point ```Ruby Parallel.map([1,2,3]) do |x| raise Parallel::Kill if x == 1# -> stop all sub-processes, killing them instantly sleep 100 # Do stuff end ``` ### Progress / ETA ```Ruby # gem install ruby-progressbar Parallel.map(1..50, progress: "Doing stuff") { sleep 1 } # Doing stuff | ETA: 00:00:02 | ==================== | Time: 00:00:10 ``` Use `:finish` or `:start` hook to get progress information. - `:start` has item and index - `:finish` has item, index, and result They are called on the main process and protected with a mutex. (To just get the index, use the more performant `Parallel.each_with_index`) ```Ruby Parallel.map(1..100, finish: -> (item, i, result) { ... do something ... }) { sleep 1 } ``` Set `finish_in_order: true` to call the `:finish` hook in the order of the input (will take longer to see initial output). ```Ruby Parallel.map(1..9, finish: -> (item, i, result) { puts "#{item} ok" }, finish_in_order: true) { sleep rand } ``` ### Worker number Use `Parallel.worker_number` to determine the worker slot in which your task is running. ```Ruby Parallel.each(1..5, in_processes: 2) { |i| puts "Item: #{i}, Worker: #{Parallel.worker_number}" } Item: 1, Worker: 1 Item: 2, Worker: 0 Item: 3, Worker: 1 Item: 4, Worker: 0 Item: 5, Worker: 1 ``` ### Dynamically generating jobs Example: wait for work to arrive or sleep ```ruby queue = [] Thread.new { loop { queue << rand(100); sleep 2 } } # job producer Parallel.map(Proc.new { queue.pop }, in_processes: 3) { |f| f ? puts("#{f} received") : sleep(1) } ``` ### Processes vs Wire serializer: security risk for hardened environments Worker processes talk to the parent over an anonymous pipe using `Marshal` by default. If you've hardened your host against `ptrace`/`/proc//mem` access (e.g. `ptrace_scope >= 2`) and want to also close the `/proc//fd/` pipe-reopen vector, use the HMAC serializer. It length-prefixes and HMAC-SHA256 signs each message with a per-worker secret generated before `fork`, so a same-UID attacker that reopens the pipe can't inject a forged `Marshal` payload into the parent (which would be RCE). Raises `SecurityError` on mismatch (not a `StandardError`). ```ruby Parallel.map(items, in_processes: 2, serializer: Parallel::Serializer::Hmac.new) { ... } ``` Tips ==== - [Benchmark/Test] Disable threading/forking with `in_threads: 0` or `in_processes: 0`, to run the same code with different setups - [Isolation] Do not reuse previous worker processes: `isolation: true` - [Stop all processes with an alternate interrupt signal] `'INT'` (from `ctrl+c`) is caught by default. Catch `'TERM'` (from `kill`) with `interrupt_signal: 'TERM'` - [Process count via ENV] `PARALLEL_PROCESSOR_COUNT=16` will use `16` instead of the number of processors detected. This is used to reconfigure a tool using `parallel` without inserting custom logic. - [Process count] `parallel` uses a number of processors seen by the OS for process count by default. If you want to use a value considering CPU quota, please add `concurrent-ruby` to your `Gemfile`. TODO ==== - Replace Signal trapping with simple `rescue Interrupt` handler Authors ======= ### [Contributors](https://github.com/grosser/parallel/graphs/contributors) - [Przemyslaw Wroblewski](https://github.com/lowang) - [TJ Holowaychuk](http://vision-media.ca/) - [Masatomo Nakano](https://github.com/masatomo) - [Fred Wu](http://fredwu.me) - [mikezter](https://github.com/mikezter) - [Jeremy Durham](http://www.jeremydurham.com) - [Nick Gauthier](http://www.ngauthier.com) - [Andrew Bowerman](http://andrewbowerman.com) - [Byron Bowerman](http://blog.bm5k.com/) - [Mikko Kokkonen](https://github.com/mikian) - [brian p o'rourke](https://github.com/bpo) - [Norio Sato] - [Neal Stewart](https://github.com/n-time) - [Jurriaan Pruis](https://github.com/jurriaan) - [Rob Worley](https://github.com/robworley) - [Tasveer Singh](https://github.com/tazsingh) - [Joachim](https://github.com/jmozmoz) - [yaoguai](https://github.com/yaoguai) - [Bartosz Dziewoński](https://github.com/MatmaRex) - [yaoguai](https://github.com/yaoguai) - [Guillaume Hain](https://github.com/zedtux) - [Adam Wróbel](https://github.com/amw) - [Matthew Brennan](https://github.com/mattyb) - [Brendan Dougherty](https://github.com/brendar) - [Daniel Finnie](https://github.com/danfinnie) - [Philip M. White](https://github.com/philipmw) - [Arlan Jaska](https://github.com/ajaska) - [Sean Walbran](https://github.com/seanwalbran) - [Nathan Broadbent](https://github.com/ndbroadbent) - [Yuki Inoue](https://github.com/Yuki-Inoue) - [Takumasa Ochi](https://github.com/aeroastro) - [Shai Coleman](https://github.com/shaicoleman) - [Earlopain](https://github.com/Earlopain) [Michael Grosser](http://grosser.it) michael@grosser.it License: MIT ## 2. Official Technical Reference & Guides (grosser/website) # Karmada Docs & Website This repo contains the source code of [Karmada website](https://karmada.io/) and all of the docs for Karmada. It's built by [Docusaurus](https://docusaurus.io/), a modern static website generator. - [Karmada website](https://karmada.io/) - [Karmada docs](https://karmada.io/docs/) - [Karmada Blog](https://karmada.io/blog/) Welcome to join us and you are more than appreciated to contribute! ## Website Preview If you only update the docs, you do not need to preview the website locally. We will create a preview link for you when you submit a pull request (PR). In an open PR, you can find the preview link in the comment of the PR. Check [Open PRs List](https://github.com/karmada-io/website/pulls). ## Run with Node.js If you have the Node.js environment, you can run the website locally. ```bash # Clone the repo, or your own fork git clone https://github.com//karmada-website.git # Install dependencies cd karmada-website npm install # Start the site yarn run start ``` ## Run with Docker If you do not want to install the Node.js environment, you can consider using Docker instead. ```bash make serve ``` With just one command, you can run it locally and then access to preview the website. ## Local Development If you want to contribute to the website, firstly, it is essential to learn some basic knowledge of [Docusaurus](https://docusaurus.io/). ### Run the development server To start a local development server, please run: ```bash yarn run start ``` This command starts a local development server and opens a browser window at . Most changes are automatically reflected live without the need to restart the server. ### Build ```bash yarn run build ``` This command generates static content into the `build` directory and can be served using any static contents hosting service. ### Deployment ```bash GIT_USER= USE_SSH=true yarn deploy ``` If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the `gh-pages` branch.