, its 403): no attempt is counted, and * that language is passed over for the rest of the run while the others go on; a run left with only * paused languages ends as the cloud's wait does (ruling R63); * - a redirect to the same page on another scheme, host or port (http to https, a www. host, a port), or * to the same address (the same Pages::key(): only its query dropped or changed outside the variables * that pick content on plain permalinks, its trailing slash changed, or itself: a host that strips * query strings, so the signed value never arrives; a loop), is a failed attempt, never "gone", in * every tier (ruling R54 as amended and refined); * - a redirect to a sign-in page (wp-login.php, WooCommerce's account page, or what the filter * langferry_sign_in_redirect says): the pair is set aside for 7 days at once, reason refused. It * counts toward no pause, and the run goes on to the next pair (ruling R55); * - a page asked for (tier 2) that answers 404, 410 or a redirect to another page is gone (a deleted * post, a changed permalink, /es/?p=5 to /es/?p=7 on plain permalinks; a firewall's challenge page * too): its row is deleted, and the run stops (ruling R41); * - otherwise a failed attempt, and the run stops: cache (the render didn't run), error (a status * that isn't 200, or a read timeout whose render hadn't recorded), unreachable (no answer) or * refused (it rendered and nothing more was translated, or the address wasn't one to ask for). The * third in a row sets the pair aside for 7 days. No answer from a language's own host (sub-domain * and custom-domain modes) counts against the language instead: the third in a row sets the whole * language aside for 7 days. * A run stops after a read timeout, so the render can finish with no second request in flight. With no * source of randomness for the request's value, nothing is cleared, asked or counted: the run stops, * system, as for a condition of the whole site. * * The worker pauses for a day (some hosts block a site's requests to itself) after three runs in a * row whose every request to the site's own host failed (no answer, 401, 403, 429 or a 5xx), or after * failed attempts at 10 different pages of the site's own host in a row with nothing done, progressing * or rendered between them (ruling R40: a page cache answering every page, a redirect to https). A * 404, a 410 or a redirect shows the site answers itself: never a failed run, but a failed attempt at * the page and one of the 10 pages. Not so a page asked for (tier 2) that is gone (ruling R41), nor a * redirect to a sign-in page, which sets the page aside at once and counts toward neither pause (ruling * R55). A redirect to the same page counts in every tier, tier 2 included (ruling R54 as amended). A * read timeout is never a failure of the site: the connection was accepted, and a big page's first * render can take longer than the timeout (ruling R45). People's visits still translate meanwhile, and * search engines translate as they did before the worker existed (is_running()). */ final class Queue { /** The cron event, and its interval. */ const HOOK = 'langferry_queue'; const INTERVAL = 'langferry_five_minutes'; const EVERY = 300; /** The hourly event (Plugin): when it is this late too, WP-Cron isn't firing (ruling R39). */ const HOURLY = 'langferry_hourly'; const HOURLY_OVERDUE = 7200; /** The run lock (an option, Pages::lock()) and how long it holds without being renewed. */ const LOCK = 'langferry_queue_lock'; const LOCK_SECONDS = 60; /** * The worker's state: paused_until, failed_runs, failed_pages (the pages that failed in a row, as * hashes of "lang path") and host_errors (language => no answers in a row). */ const STATE = 'langferry_queue'; /** Failed attempts in a row that set a pair aside; no answers in a row that set a language aside; failed runs in a row that pause the worker. */ const IN_A_ROW = 3; /** Different pages that failed in a row, which pause the worker too (ruling R40). */ const PAGES_IN_A_ROW = 10; const SET_ASIDE = 604800; // 7 days. const PAUSE = 86400; // 1 day. const OVERDUE = 3600; // An event this late means cron isn't running. const TIMEOUT = 20; /** How long `wp langferry translate` gives a render that outlasted its request to record (ruling R45). */ const LATE = 20; /** A step that is a failed attempt, by its outcome. */ const FAILED = array( 'cache', 'unreachable', 'error', 'refused', 'signin' ); /** * One run of the worker (the cron event): at most $max_pairs pairs, and no request started after * $max_seconds. The unit is a pair, a page in one language: one asked for again in the same run * (more translated, not finished yet) counts once, and the time limit bounds the requests. * * @return array{done: int, attempts: int, stopped: string, languages_paused?: string[]} done: pairs * completed; attempts: pairs tried, each once; stopped: why the run ended (inactive, paused, * paced, cloud, system, locked, failed, gone, timeout, empty, pairs or time; system: PHP has * no source of randomness); languages_paused: only when the run ended cloud because the only * languages left were paused by the cloud, their codes (ruling R63). */ public static function run( int $max_pairs = 3, int $max_seconds = 20 ): array { return self::walk( $max_pairs, $max_seconds, '', true, null ); } /** * `wp langferry translate`: the same order and the same step as the worker, without a run's time * limit. It goes on while the worker is paused (it is what the settings page suggests then), and * its failures don't count toward the pause. A failed attempt is recorded as the worker's are, and * that pair is skipped for the rest of the run (ruling R35); 3 failures of the site's own host in a * row (as for the pause: no answer, 401, 403, 429 or a 5xx) stop it (`site`). After a read timeout * it gives the render up to 20 more seconds to record before its next request (ruling R45). * * @param int $limit At most this many pairs, each once. * @param string $lang Only this language ('' for all). * @param callable|null $each Called after each request with the pair and the step's result * (outcome, status, url, requested, timeout). * @return array{done: int, attempts: int, stopped: string, languages_paused?: string[]} As run(); * stopped is also site, never failed, gone, timeout, paused or time. */ public static function pretranslate( int $limit, string $lang = '', ?callable $each = null ): array { return self::walk( $limit, 0, $lang, false, $each ); } /** * Whether the worker is at work: the site is connected with languages, the worker isn't paused, * and cron fires. Cron doesn't fire when the worker's event is more than an hour late, the hourly * event more than 2 hours late, or neither is scheduled (ruling R39: WP-Cron off without a system * cron, or a host that blocks the site's requests to itself). While it isn't, search engines * translate new text as they did before the worker existed. */ public static function is_running(): bool { $now = self::now(); $next = wp_next_scheduled( self::HOOK ); $hourly = wp_next_scheduled( self::HOURLY ); if ( ! self::can_run( $now ) || ( false === $next && false === $hourly ) ) { return false; } return ! ( false !== $next && $next < $now - self::OVERDUE ) && ! ( false !== $hourly && $hourly < $now - self::HOURLY_OVERDUE ); } /** * running (at work, with its event scheduled), idle (nothing to do, or it can't run), or * paused_loopback (the site's requests to itself kept failing). */ public static function state(): string { if ( self::paused( self::now() ) ) { return 'paused_loopback'; } return self::is_running() && false !== wp_next_scheduled( self::HOOK ) ? 'running' : 'idle'; } /** * Schedule the event while the worker can run and the order has work, and remove it otherwise. * Decided at the end of each run, hourly, on activation and when the settings are updated, never * on a visitor's request. */ public static function schedule(): void { $now = self::now(); $want = self::can_run( $now ) && ( Priority::has_more( $now ) || null !== Priority::next( $now ) ); $next = wp_next_scheduled( self::HOOK ); if ( $want && false === $next ) { wp_schedule_event( $now, self::INTERVAL, self::HOOK ); } elseif ( ! $want && false !== $next ) { wp_clear_scheduled_hook( self::HOOK ); } } /** * The worker's interval, every five minutes (cron_schedules). * * @param mixed $schedules * @return array */ public static function cron_schedules( $schedules ) { $schedules = is_array( $schedules ) ? $schedules : array(); $schedules[ self::INTERVAL ] = array( 'interval' => self::EVERY, // Translations load at init; a plugin may ask for the schedules earlier. 'display' => did_action( 'init' ) ? __( 'Every five minutes (Langferry)', 'langferry' ) : 'Every five minutes (Langferry)', ); return $schedules; } /** * The pairs, one after the other, until a limit, a stop or the end of the order. * * @param bool $worker The cron worker (true) or the CLI: only the worker stops while paused and * counts failed runs and pages. */ private static function walk( int $max_pairs, int $max_seconds, string $lang, bool $worker, ?callable $each ): array { $start = self::now(); $out = array( 'done' => 0, 'attempts' => 0, 'stopped' => '', ); if ( ! Settings::is_active() ) { $out['stopped'] = 'inactive'; } elseif ( $worker && self::paused( $start ) ) { $out['stopped'] = 'paused'; } else { $out['stopped'] = (string) self::blocked( '' ); } if ( '' !== $out['stopped'] ) { self::schedule(); return $out; } $held = self::lock_value( $start ); if ( ! Pages::lock( self::LOCK, $held, $start ) ) { $out['stopped'] = 'locked'; return $out; } $own = array( 'ok' => 0, 'failed' => 0, ); $pages = array(); // The worker's: what each request to the site's own host says of it (count_run()). $tried = array(); // The pairs tried, "lang path": one asked for again counts once. $skip = array(); // The CLI's: the pairs that failed in this run, passed over (ruling R35). Never the worker's. $skip_langs = array(); // The languages the cloud paused, passed over for the rest of the run (ruling R63). $strike = 0; // The CLI's: failures of the site's own host in a row. try { while ( '' === $out['stopped'] ) { $now = self::now(); if ( $max_seconds > 0 && $now - $start >= $max_seconds ) { $out['stopped'] = 'time'; break; } $pair = Priority::next( $now, $skip, $lang, $skip_langs ); if ( null === $pair ) { if ( ! $skip && Priority::has_more( $now, $lang, $skip_langs ) ) { continue; // A window of the sitemap with nothing waiting: the next one. } // With a skip list, next() has read on to the end. Left with only the languages the // cloud paused, the run waits for the cloud, and says which ones (ruling R63). $out['stopped'] = $skip_langs ? 'cloud' : 'empty'; if ( $skip_langs ) { $out['languages_paused'] = $skip_langs; } break; } $id = $pair['lang'] . ' ' . $pair['path']; if ( ! isset( $tried[ $id ] ) && count( $tried ) >= $max_pairs ) { $out['stopped'] = 'pairs'; break; } $blocked = self::blocked( $pair['lang'], 1 === (int) $pair['tier'] ); if ( 'language' === $blocked ) { $skip_langs[] = $pair['lang']; // Passed over with nothing counted; the other languages go on. continue; } if ( null !== $blocked ) { $out['stopped'] = $blocked; break; } // Renewed before each request: a run that lost its lock (it ran past its 60 seconds and // another run took over) never asks for another page. $renewed = self::lock_value( $now, $held ); if ( ! Pages::relock( self::LOCK, $held, $renewed ) ) { $out['stopped'] = 'locked'; return $out; } $held = $renewed; $step = self::step( $pair, $worker ); if ( 'system' === $step['outcome'] ) { // No source of randomness: a condition of the whole site, like a cloud stop. Nothing was // asked or written, and no pair is counted or set aside. $out['stopped'] = 'system'; break; } Priority::forget_memo(); // The render wrote the pages table in a request of its own. $tried[ $id ] = true; $out['attempts'] = count( $tried ); $failed = in_array( $step['outcome'], self::FAILED, true ); $signin = 'signin' === $step['outcome']; // Set aside at once, and counted by neither pause (ruling R55). $gone = 'gone' === $step['outcome']; $own_request = $step['own'] && $step['requested']; // Only a failed attempt counts toward the pause (ruling R36), and never a read timeout: the // connection was accepted (ruling R45). $site_failed = $failed && ! $signin && $own_request && ! $step['timeout'] && self::refused_by_site( $step['status'] ); if ( $own_request && ( 200 === $step['status'] || in_array( $step['outcome'], array( 'done', 'progress' ), true ) ) ) { ++$own['ok']; } elseif ( $site_failed ) { ++$own['failed']; } if ( $own_request ) { $strike = $site_failed ? $strike + 1 : 0; // Ruling R40: a page done, progressing or rendered starts the count of failed pages // again; a failed attempt adds its page, a read timeout neither. if ( in_array( $step['outcome'], array( 'done', 'progress', 'refused' ), true ) ) { $pages[] = true; } elseif ( $failed && ! $signin && ! $step['timeout'] ) { $pages[] = $id; } } if ( null !== $each ) { $each( $pair, $step ); } if ( 'done' === $step['outcome'] ) { ++$out['done']; } elseif ( in_array( $step['outcome'], array( 'paced', 'cloud' ), true ) ) { $out['stopped'] = $step['outcome']; } elseif ( 'language' === $step['outcome'] ) { $skip_langs[] = $pair['lang']; // The cloud paused it during this request: no attempt; the run goes on. } elseif ( $signin ) { // Set aside already, so next() can't give it again: the run goes on to the next pair, in the // worker and the CLI alike (it was answered, so no read timeout to wait for). Not on the skip // list, which would make the worker's next() read window after window in one call, past the // run's time check (ruling R28). continue; } elseif ( $worker && ( $failed || $gone ) ) { $out['stopped'] = $gone ? 'gone' : 'failed'; // The next attempt waits for the next run. } elseif ( $failed || $gone ) { $skip[ $id ] = true; if ( $strike >= self::IN_A_ROW ) { $out['stopped'] = 'site'; } } if ( $worker && $step['timeout'] && in_array( $out['stopped'], array( '', 'failed' ), true ) ) { $out['stopped'] = 'timeout'; // Its render may still be running: no second request now. } } } finally { Pages::unlock( self::LOCK, $held ); // Only its own: a lock another run took over stays. } if ( $worker ) { self::count_run( $own['ok'], $own['failed'], $pages ); } self::schedule(); return $out; } /** * One pair: clear, ask, read the row, and count what happened. * * @param array{lang: string, path: string, tier: int, label: string} $pair * @param bool $worker The cron worker, or * the CLI, which waits for a render that outlasted its request. * @return array{outcome: string, status: int, url: string, own: bool, requested: bool, timeout: bool} */ private static function step( array $pair, bool $worker ): array { $router = Router::instance(); $lang = $pair['lang']; $source = home_url( $pair['path'] ); $target = (string) $router->translate_url( $source, $lang ); $host = self::host( $target ); $own = $host === $router->home_host(); $result = array( 'outcome' => '', 'status' => 0, 'url' => $target, 'own' => $own, 'requested' => false, 'timeout' => false, ); // Only the pair's translated address, on the site's own host or the language's: never another // site (a filter can send home_url() elsewhere), and never the original language's page. $scheme = strtolower( (string) wp_parse_url( $target, PHP_URL_SCHEME ) ); if ( $target === $source || ! in_array( $scheme, array( 'http', 'https' ), true ) || ( ! $own && $host !== $router->lang_host( $lang ) ) ) { $result['outcome'] = 'refused'; self::failed( $pair, 'refused' ); return $result; } // The request's random part, new each time, before anything is cleared or asked for. With no source // of randomness (random_bytes() throws) nothing is cleared, asked or written: the run stops, system // (PRIO-WORK-019). try { // Unqualified on purpose: the tests' hook (tests/fixtures/random-bytes-seam.php). Not secret: the HMAC protects the value. $random = bin2hex( random_bytes( 6 ) ); } catch ( \Exception $e ) { $result['outcome'] = 'system'; return $result; } $main = 1 === (int) $pair['tier']; $before = Pages::get( $pair['path'], $lang ); Cache_Purge::url( $target ); $asked = self::now(); $start = gmdate( 'Y-m-d H:i:s', $asked ); // Signed for this request and its start, in its langferry_fetch value: never a header, which // something between the worker and PHP may strip (ruling R56 as amended). A main page's says so // (ruling R66). With no valid secret nothing is signed: the request still carries its random part, // so caches pass it, and it renders as a script's (cache). $fetch = Fetch_Signature::sign( $random, $asked, $main ); $response = wp_remote_get( self::with_fetch_arg( $target, '' === $fetch ? $random : $fetch ), array( 'timeout' => self::TIMEOUT, 'redirection' => 0, 'sslverify' => apply_filters( 'https_local_ssl_verify', false ), // phpcs:ignore WordPress.NamingConventions.PrefixAllGlobals.NonPrefixedHooknameFound -- WordPress's own filter for requests to the site itself. 'cookies' => array(), 'headers' => array( 'User-Agent' => 'Langferry-Pretranslate/' . LANGFERRY_VERSION ), ) ); $result['requested'] = true; $answered = ! is_wp_error( $response ); $result['status'] = $answered ? (int) wp_remote_retrieve_response_code( $response ) : 0; $result['timeout'] = ! $answered && self::read_timeout( $response ); $row = Pages::get( $pair['path'], $lang ); // Read even after a timeout: the render may still have recorded it. $rendered = null !== $row && $row['updated_at'] >= $start; if ( $result['timeout'] && ! $rendered && ! $worker ) { $row = self::wait_for_render( $pair, $start ); $rendered = null !== $row && $row['updated_at'] >= $start; } $blocked = self::blocked( $lang, $main, true ); // The render may have just been paced, or its language paused. if ( ! $own ) { self::host_answered( $lang, $answered || $result['timeout'], $pair ); // A read timeout: the host accepted the connection. } // The page itself only when the site served it (200), or its render outlasted the request and // recorded: a maintenance mode's 503 page renders through the theme too (rulings R38, R45). $served = 200 === $result['status'] || ( $result['timeout'] && $rendered ); $more = $rendered && $row['translated_chars'] > ( null === $before ? 0 : $before['translated_chars'] ); $kind = $answered ? self::redirect_kind( $target, $response, $lang ) : ''; if ( $served && null !== $row && 0 === $row['pending'] ) { $result['outcome'] = 'done'; Cache_Purge::url( $target ); // Visitors and crawlers get the finished page. if ( $row['attempts'] > 0 || '' !== $row['reason'] ) { Pages::clear_attempts( $pair['path'], $lang ); } } elseif ( null !== $blocked ) { $result['outcome'] = $blocked; // It waits; not an attempt. } elseif ( $served && $more ) { $result['outcome'] = 'progress'; // The render reset the count. } elseif ( 'sign_in' === $kind ) { // A redirect to a sign-in page: visitors can't read the page, so it is set aside at once for // 7 days, counting toward no pause (ruling R55). $result['outcome'] = 'signin'; self::failed( $pair, 'refused', 1 ); } elseif ( 2 === $pair['tier'] && '' === $kind && self::gone( $result['status'] ) ) { // A page a person asked for that isn't there any more: off the queue, not set aside to come // back every 7 days ahead of tier 3 (ruling R41). Tiers 1 and 3 are pages WordPress lists. // A redirect to the same page, on any scheme, host or port, isn't this (ruling R54 as amended and refined). $result['outcome'] = 'gone'; Pages::forget_pair( $pair['path'], $lang ); } elseif ( ! $answered && ! $result['timeout'] ) { $result['outcome'] = 'unreachable'; if ( $own ) { self::failed( $pair, 'unreachable' ); } } else { if ( $result['timeout'] ) { $result['outcome'] = $rendered ? 'refused' : 'error'; // A late render that translates more resets the count (Pages::record). } elseif ( 200 !== $result['status'] ) { $result['outcome'] = 'error'; } elseif ( ! $rendered ) { $result['outcome'] = 'cache'; } else { $result['outcome'] = 'refused'; } self::failed( $pair, $result['outcome'] ); } // The original page's hreflang lists a translation only while it is listed: a render that changed // that clears the original-language address too, each tool as exactly as it can (ruling R64). if ( $rendered && Pages::listed( $before ) !== Pages::listed( $row ) ) { Cache_Purge::url( $source ); } return $result; } /** * The address with the throwaway query argument (ruling R43) and this request's signed value * (Fetch_Signature::sign(): 12 random hex digits, new for each request, the time, the tier's flag * and the signature; the random part alone when nothing could be signed), so a page cache in front * of PHP passes it through. The router drops it again. */ private static function with_fetch_arg( string $url, string $fetch ): string { $parts = explode( '#', $url, 2 ); return $parts[0] . ( false === strpos( $parts[0], '?' ) ? '?' : '&' ) . Router::FETCH_ARG . '=' . $fetch; } /** * Whether a request failed reading its answer: the connection was accepted, but the page took * longer than the timeout (cURL error 28, "Operation timed out after …"; "fsocket timed out" * without cURL). A connection that couldn't be made in time ("Connection timed out", "Resolving * timed out") is no answer. * * @param mixed $response What wp_remote_get() returned. */ private static function read_timeout( $response ): bool { return is_wp_error( $response ) && 1 === preg_match( '/\bcURL error 28: Operation timed out\b|^fsocket timed out\b/i', $response->get_error_message() ); } /** * `wp langferry translate`, after a read timeout: the pair's row once a second until the render * records it (written at or after $start), for up to 20 seconds, so it asks for no other page while * the render still runs. The last row read, or null. */ private static function wait_for_render( array $pair, string $start ): ?array { $until = self::now() + self::LATE; $row = null; while ( self::now() < $until ) { self::sleep( 1 ); $row = Pages::get( $pair['path'], $pair['lang'] ); if ( null !== $row && $row['updated_at'] >= $start ) { break; } } return $row; } /** * Wait $seconds. The filter langferry_queue_sleep is for the plugin's own tests: false skips the * real wait (they move the clock, langferry_queue_now, themselves). */ private static function sleep( int $seconds ): void { if ( false !== apply_filters( 'langferry_queue_sleep', true, $seconds ) ) { sleep( $seconds ); } } /** * Why pre-translation's render can't reach the cloud now, so the worker waits rather than count an * attempt; or null. The global stops first: paced, the plan's pace says not yet, for everyone * (people's) or for pre-translation of the next pair's tier (ruling R67 as amended: a main page's * flag apart from the other pages'; a search engine's pace never holds the worker); cloud, its * backoff, the allowance used up, the key refused or page translation not in the plan. Then * language: only $lang is paused by the cloud, and the run passes it over (ruling R63). * * @param string $lang The next pair's language, or '' before a pair is known. * @param bool|null $main Whether the next pair is a main page (tier 1); null before a pair is known, * when people's pace counts, and pre-translation's only when both of its * flags are set (no tier could go). * @param bool $fresh Read past this request's cache: the render just asked for may have set them. */ private static function blocked( string $lang, ?bool $main = null, bool $fresh = false ): ?string { $set = static function ( $name ) use ( $fresh ) { return $fresh ? self::fresh_transient( $name ) : false !== get_transient( $name ); }; // Before a pair is known, both pre-translation flags together mean no tier can go. $pretranslate = null === $main ? $set( Translator::PACED_PRETRANSLATE_MAIN ) && $set( Translator::PACED_PRETRANSLATE ) : $set( $main ? Translator::PACED_PRETRANSLATE_MAIN : Translator::PACED_PRETRANSLATE ); if ( $set( Translator::PACED_PEOPLE ) || $pretranslate ) { return 'paced'; } $conn = Settings::connection(); if ( ! empty( $conn['api_key_invalid'] ) || ( $fresh ? self::fresh_transient( 'langferry_cloud_backoff' ) : Cloud::in_backoff() ) || $set( 'langferry_quota_exceeded' ) || $set( Translator::PLAN_PAUSE . 'page' ) ) { return 'cloud'; } return '' !== $lang && $set( 'langferry_pause_' . $lang ) ? 'language' : null; } /** * Whether a transient is set, read from where it is stored: another request (the render the * worker asked for) may have set it after this one last read it. */ private static function fresh_transient( string $name ): bool { if ( wp_using_ext_object_cache() ) { wp_cache_get( $name, 'transient', true ); // Fetched from the store, which refreshes this request's copy. } else { $options = array( '_transient_' . $name, '_transient_timeout_' . $name ); $missing = wp_cache_get( 'notoptions', 'options' ); if ( is_array( $missing ) ) { wp_cache_set( 'notoptions', array_diff_key( $missing, array_flip( $options ) ), 'options' ); } foreach ( $options as $option ) { wp_cache_delete( $option, 'options' ); } } return false !== get_transient( $name ); } /** * A failed attempt at the pair, at now: the $limit-th in a row (the third) sets it aside for 7 days. */ private static function failed( array $pair, string $reason, int $limit = self::IN_A_ROW ): void { $now = self::now(); Pages::failed( $pair['path'], $pair['lang'], $reason, $limit, gmdate( 'Y-m-d H:i:s', $now ), gmdate( 'Y-m-d H:i:s', $now + self::SET_ASIDE ) ); } /** * A request to a language's own host: no answer is one more in a row, and the third sets the * language aside for 7 days; any answer starts the count again. */ private static function host_answered( string $lang, bool $answered, array $pair ): void { $state = self::state_option(); $errors = (int) ( $state['host_errors'][ $lang ] ?? 0 ); if ( $answered && 0 === $errors ) { return; } $errors = $answered ? 0 : $errors + 1; if ( $errors >= self::IN_A_ROW ) { Priority::set_language_aside( $lang, self::now() + self::SET_ASIDE, 'unreachable', $pair['label'] ); $errors = 0; } if ( $errors > 0 ) { $state['host_errors'][ $lang ] = $errors; } else { unset( $state['host_errors'][ $lang ] ); } self::save_state( $state ); } /** * After a worker's run: one page served by the site's own host starts the count of failed runs * again; a run whose every request to it was a failed attempt the site refused (refused_by_site()) * is one more, and the third pauses the worker for a day. Any other run (paced, nothing to do, a * 404, a read timeout) leaves it as it is. * * $pages, in order, says what each request to the site's own host showed of the pages (ruling * R40): true for a page done, progressing or rendered, which starts the count of failed pages * again, or the pair ("lang path") a failed attempt was at. Failed attempts at 10 different pages * in a row pause the worker for a day too. * * @param array $pages */ private static function count_run( int $ok, int $failed, array $pages ): void { $state = self::state_option(); foreach ( $pages as $page ) { if ( true === $page ) { $state['failed_pages'] = array(); } elseif ( ! in_array( sha1( $page ), $state['failed_pages'], true ) ) { $state['failed_pages'][] = sha1( $page ); } } $pause = count( $state['failed_pages'] ) >= self::PAGES_IN_A_ROW; if ( $ok > 0 ) { $state['failed_runs'] = 0; } elseif ( $failed > 0 ) { ++$state['failed_runs']; $pause = $pause || $state['failed_runs'] >= self::IN_A_ROW; } if ( $pause ) { $state['failed_runs'] = 0; $state['failed_pages'] = array(); $state['paused_until'] = self::now() + self::PAUSE; } self::save_state( $state ); } /** * What a 3xx answer is (rulings R54, R55): 'sign_in' when it sends the visitor to a sign-in page, * 'site' when it is the same page: the same path (trailing slash ignored, percent-decoded) on another * scheme, host or port (R54), or on the same one the same Pages::key() (trailing slash ignored): only * the query dropped or changed outside the variables that pick content on plain permalinks, or the * address itself (R54 as amended and refined); and '' otherwise (another page: a moved page, a * firewall's challenge page, /es/?p=7 for /es/?p=5), including any answer that isn't a redirect. * * @param string $target The address asked for. * @param mixed $response The HTTP response. * @param string $lang The pair's language. */ private static function redirect_kind( string $target, $response, string $lang ): string { $status = (int) wp_remote_retrieve_response_code( $response ); if ( $status < 300 || $status > 399 ) { return ''; } $location = wp_remote_retrieve_header( $response, 'location' ); if ( is_array( $location ) ) { $location = reset( $location ); } $location = is_string( $location ) ? trim( $location ) : ''; if ( '' === $location ) { return ''; } $location = \WP_Http::make_absolute_url( $location, $target ); if ( self::sign_in( $location, $lang ) ) { return 'sign_in'; } $from = wp_parse_url( $target ); $to = wp_parse_url( $location ); if ( ! is_array( $from ) || ! is_array( $to ) ) { return ''; } $where = static function ( array $parts ): array { $scheme = strtolower( (string) ( $parts['scheme'] ?? '' ) ); $port = isset( $parts['port'] ) ? (int) $parts['port'] : ( 'https' === $scheme ? 443 : 80 ); return array( $scheme, strtolower( (string) ( $parts['host'] ?? '' ) ), $port ); }; $path = static function ( array $parts ): string { return untrailingslashit( rawurldecode( (string) ( $parts['path'] ?? '' ) ) ); }; if ( $where( $from ) !== $where( $to ) ) { // The same path on another scheme, host or port is the same page (ruling R54). return $path( $from ) === $path( $to ) ? 'site' : ''; } // On the same one, the same page is the same Pages::key(), trailing slash ignored (R54 as amended // and refined): the request always carries langferry_fetch, so a host that strips or rewrites query // strings answers this way, or a loop does. The key keeps the query variables that pick content on // plain permalinks, where every page shares a path: /es/?p=5 to /es/?p=7 is another page. $key = static function ( string $url ): string { $parts = explode( '?', Pages::key( $url ), 2 ); return untrailingslashit( rawurldecode( $parts[0] ) ) . ( isset( $parts[1] ) ? '?' . $parts[1] : '' ); }; return $key( $target ) === $key( $location ) ? 'site' : ''; } /** * Whether an address is a sign-in page (ruling R55): WordPress's own, any wp-login.php, * WooCommerce's account page or a page under it, or what the filter langferry_sign_in_redirect says. * * @param string $location The absolute address a redirect leads to. * @param string $lang The pair's language: a language prefix or translated slug is read away. */ private static function sign_in( string $location, string $lang ): bool { $login = wp_login_url(); $login_key = Pages::key( $login ); // Pages::key() twice: as the site serves the address, and with a language prefix or translated slug read away. $key = Pages::key( $location, $lang ); $by = ! in_array( $login_key, array( '', '/' ), true ) && ( Pages::key( $location ) === $login_key || $key === $login_key ); // A login key of '/' would match every redirect to the home page. $is = $by || self::is_login_page( $location, $login ) || '/wp-login.php' === substr( untrailingslashit( (string) wp_parse_url( $location, PHP_URL_PATH ) ), -13 ); $account_id = ! $is && function_exists( 'wc_get_page_id' ) ? (int) wc_get_page_id( 'myaccount' ) : 0; if ( $account_id > 0 ) { $account = untrailingslashit( Pages::key( (string) get_permalink( $account_id ) ) ); $is = '' !== $account && ( untrailingslashit( $key ) === $account || 0 === strpos( $key, $account . '/' ) ); } return (bool) apply_filters( 'langferry_sign_in_redirect', $is, $location ); } /** * A login URL the key can't tell from the home page (a security plugin's `/?my-login`, or a login at * `/`): the Location has the same path and carries every query argument name of the login URL, values * ignored. A login URL with no argument to tell it apart is never matched this way (ruling R55). The * paths are compared in the original language, through the router's own Router::translate_url(), so * a language prefix (sub-directory mode's "/es/?my-login") or a translated slug is read away. */ private static function is_login_page( string $location, string $login ): bool { if ( ! in_array( Pages::key( $login ), array( '', '/' ), true ) ) { return false; } $want = array(); parse_str( (string) wp_parse_url( $login, PHP_URL_QUERY ), $want ); if ( ! $want ) { return false; } $have = array(); parse_str( (string) wp_parse_url( $location, PHP_URL_QUERY ), $have ); $router = Router::instance(); $original = (string) Settings::effective()['default_language']; $path = static function ( string $url ) use ( $router, $original ): string { return untrailingslashit( (string) wp_parse_url( (string) $router->translate_url( $url, $original ), PHP_URL_PATH ) ); }; return $path( $location ) === $path( $login ) && ! array_diff_key( $want, $have ); } /** * Whether a page asked for (tier 2) isn't there any more: a 404, a 410 or a redirect (any 3xx; the * worker follows none), ruling R41. step() first passes over a redirect to the same page or to a * sign-in page (redirect_kind()). */ private static function gone( int $status ): bool { return 404 === $status || 410 === $status || ( $status >= 300 && $status < 400 ); } /** * Whether an answer from the site's own host says it can't be reached or turns the worker away * (ruling R34): no answer (0), 401, 403, 429 or a 5xx, maintenance mode's 503 among them. A 404, a * 410 or a redirect shows the site answers itself. */ private static function refused_by_site( int $status ): bool { return 0 === $status || in_array( $status, array( 401, 403, 429 ), true ) || $status >= 500; } /** Whether the site is connected with languages and the worker isn't paused. */ private static function can_run( int $now ): bool { return Settings::is_active() && ! self::paused( $now ); } private static function paused( int $now ): bool { return self::state_option()['paused_until'] > $now; } /** * @return array{paused_until: int, failed_runs: int, failed_pages: string[], host_errors: array} */ private static function state_option(): array { $saved = get_option( self::STATE ); $saved = is_array( $saved ) ? $saved : array(); $pages = is_array( $saved['failed_pages'] ?? null ) ? array_values( array_filter( $saved['failed_pages'], 'is_string' ) ) : array(); return array( 'paused_until' => (int) ( $saved['paused_until'] ?? 0 ), 'failed_runs' => (int) ( $saved['failed_runs'] ?? 0 ), 'failed_pages' => array_slice( $pages, 0, self::PAGES_IN_A_ROW ), 'host_errors' => is_array( $saved['host_errors'] ?? null ) ? array_map( 'intval', $saved['host_errors'] ) : array(), ); } /** Autoloaded: is_running() is read on search engines' requests. */ private static function save_state( array $state ): void { update_option( self::STATE, $state, true ); } /** * The lock's value at $now: its expiry, 60 seconds on, and this run's token (kept from $held). */ private static function lock_value( int $now, string $held = '' ): string { $parts = explode( ':', $held, 2 ); $token = isset( $parts[1] ) && '' !== $parts[1] ? $parts[1] : wp_generate_password( 12, false ); return ( $now + self::LOCK_SECONDS ) . ':' . $token; } private static function host( string $url ): string { $parts = wp_parse_url( $url ); return is_array( $parts ) ? strtolower( ( $parts['host'] ?? '' ) . ( isset( $parts['port'] ) ? ':' . $parts['port'] : '' ) ) : ''; } /** * The worker's clock. The filter langferry_queue_now is for the plugin's own tests: it moves the * clock without waiting. */ private static function now(): int { return (int) apply_filters( 'langferry_queue_now', time() ); } }