i have been trying resque task running. resque seems enqueueing , running tasks; however, tasks seem "deadlocking" upon execution.
a sample error follows: (taken resque::failure output)
{"failed_at"=>"2015/07/20 22:34:46 utc", "payload"=>{"class"=>"importhawaiieventsonline", "args"=>[]}, "exception"=>"fatal", "error"=>"no live threads left. deadlock?", "backtrace"=> ["/usr/local/lib/ruby/2.1.0/open3.rb:262:in `value'", "/usr/local/lib/ruby/2.1.0/open3.rb:262:in `block in capture3'", "/usr/local/lib/ruby/2.1.0/open3.rb:199:in `popen_run'", "/usr/local/lib/ruby/2.1.0/open3.rb:93:in `popen3'", "/usr/local/lib/ruby/2.1.0/open3.rb:252:in `capture3'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/shell.rb:47:in `execute_open3'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/shell.rb:36:in `block (2 levels) in execute'", "/usr/local/lib/ruby/2.1.0/timeout.rb:76:in `timeout'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/shell.rb:35:in `block in execute'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/logger.rb:32:in `call'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/logger.rb:32:in `block in benchmark'", "/usr/local/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/logger.rb:32:in `benchmark'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/logger.rb:21:in `debug'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/shell.rb:34:in `execute'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/shell.rb:18:in `run'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/tool.rb:92:in `call'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/tool.rb:53:in `new'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/image.rb:477:in `identify'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/image.rb:198:in `validate!'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/image.rb:113:in `block in create'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/image.rb:112:in `tap'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/image.rb:112:in `create'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/image.rb:34:in `read'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/image.rb:90:in `block in open'", "/usr/local/lib/ruby/2.1.0/open-uri.rb:36:in `open'", "/usr/local/lib/ruby/2.1.0/open-uri.rb:36:in `open'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/image.rb:89:in `open'", "/srv/releases/4/app/uploaders/photo_uploader.rb:57:in `get_geometry'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/processing.rb:84:in `block in process!'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/processing.rb:76:in `each'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/processing.rb:76:in `process!'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/callbacks.rb:18:in `block in with_callbacks'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/callbacks.rb:18:in `each'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/callbacks.rb:18:in `with_callbacks'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/cache.rb:122:in `cache!'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/download.rb:72:in `download!'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/mount.rb:356:in `remote_url='", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/mount.rb:187:in `remote_photo_url='", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/orm/activerecord.rb:45:in `remote_photo_url='", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/attribute_assignment.rb:45:in `public_send'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/attribute_assignment.rb:45:in `_assign_attribute'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/attribute_assignment.rb:32:in `block in assign_attributes'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/attribute_assignment.rb:26:in `each'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/attribute_assignment.rb:26:in `assign_attributes'", "/srv/releases/4/event_jobs/generic_import.rb:45:in `event_endpoint'", "/srv/releases/4/event_jobs/generic_import.rb:113:in `save_event'", "/srv/releases/4/event_jobs/hawaiieventonline/import.rb:26:in `block (2 levels) in get_parser'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/nokogiri-1.6.6.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/nokogiri-1.6.6.2/lib/nokogiri/xml/node_set.rb:186:in `upto'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/nokogiri-1.6.6.2/lib/nokogiri/xml/node_set.rb:186:in `each'", "/srv/releases/4/event_jobs/hawaiieventonline/import.rb:24:in `each_with_index'", "/srv/releases/4/event_jobs/hawaiieventonline/import.rb:24:in `block in get_parser'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/nokogiri-1.6.6.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/nokogiri-1.6.6.2/lib/nokogiri/xml/node_set.rb:186:in `upto'", "/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/nokogiri-1.6.6.2/lib/nokogiri/xml/node_set.rb:186:in `each'", "/srv/releases/4/event_jobs/hawaiieventonline/import.rb:21:in `get_parser'", "/srv/releases/4/event_jobs/generic_import.rb:22:in `run_parse'", "/srv/releases/4/app/jobs/import_hawaiieventsonline.rb:17:in `perform'"], "worker"=> "be920dfe9b17:5637:`(queues)`", "queue"=>"`(queue)`"}, i guessing had mini_magick gem - though seems odd suggest deadlock since i'm running 1 thread. guess possibly due sheer amount of requests being made asynchronously - though don't understand mini_magick , carrierwave gems.
does have suggestions on resolve thread issue? running 17 queues 1 worker thread handling of these queues @ moment, , seems run fine - there may issue saving cloud. trying search related mini_magick/carrierwave , thread issue seems not show anything.
thanks in advance.
i'm curious, running resque-pool or other forking/threading in code? minimagick uses open3 run shell commands, , apparently has reputation not playing nice threads. luckily, minimagick allows swap out open3 posix-spawn.
i getting same error using threaded workers (resque-pool jobs) , switching posix-spawn appears have resolved error me.
Comments
Post a Comment