How to scrape a website that requires login

To scrape a website that requires login, you sign in once, keep the session the site gives you, and send that session with every later request. The session lives in cookies, so the work is getting them and keeping them fresh. There are three ways to do it: copy the cookies from your browser, script the login form with an HTTP session, or sign in with a real browser and save its state. Do it only with an account you own, on a site whose terms allow it.

Which one you need depends on how the site signs you in. A plain HTML form works with a script. A login with JavaScript checks, two-factor codes or a bot wall needs a real browser.

Pick the method by how the site signs you in

Each method below costs more effort than the one before it, so start at the top.

All four rest on the same mechanism. A Stack Overflow answer with 75 votes puts it plainly:

"Basically, when you login into a site in a normal way, you identify yourself in a unique way using your credentials, and the same identity is used thereafter for every other interaction, which is stored in cookies and headers, for a brief period of time."

Adelin, Stack Overflow, April 2020

"For a brief period of time" is the part that bites. Every method needs a way to notice an expired session and sign in again.

How String handles pages behind a login

String is the most reliable web scraping API, bypassing anti-bot, and you only pay when content comes back. For pages behind a login you own, one fact per line:

• Sign in once, reuse the login. A browser context is a saved browser profile: sign in over String's remote browser, close it, and every later session with the same contextId starts with the login, cookies and site storage in place.
• Read in parallel from one login. Open as many read-only sessions (persist=false) as you need at once from a saved profile, alongside the one session that saves changes.
• The browser is the one you already script. Connect Playwright or Puppeteer to String's remote Chrome over a CDP WebSocket (browser docs), so your login code does not change.
• Challenge pages get solved before your page loads them. String can solve CAPTCHAs and challenge pages inside a browser session (captcha docs), which is where scripted logins usually stall.
• The exit country stays fixed. Send the same countryCode on every connection, and a restored login comes from the country it was made in (contexts docs).
• Login flows are enabled on request. Some destinations and POST requests to sign-in paths need approval first; String's docs ask you to email support with the paths you need (access control).
• The same API is measured in public. String returned content on 98.6% of 500 requests to 100 bot-protected pages in the October 30, 2026 run, the highest of 17 APIs (benchmark).

Where String fits

• For: teams that read data from their own accounts, or accounts they are licensed to use, on sites with bot checks at login, and who would rather not host browsers and proxies.
• Not for: accounts you do not own, logins made with fake identities, or a site that sells the same data through an official API.
• Price: browser sessions bill $0.12 an hour plus $5.00 per GB on standard proxies or $10.00 per GB on premium, on every plan (pricing). The free tier gives $5 of credit, and refused connections are not billed.
• Integrations: Playwright and Puppeteer over CDP, a REST API for public pages, and an MCP server for agents.
• Limits: 25 saved profiles per organization by default, 10 saving sessions at once, and a profile expires after 30 days without use (contexts docs).

How to scrape behind a login, step by step

Step 1: Check whether you need to log in at all

Open the page in a private window first. Many sites show product pages, listings and profiles to signed-out visitors, and scraping the logged-out view keeps you clear of the account terms covered below. If the site offers an official API or a data export for your own account, use that.

If the data is public, String's fetch endpoint returns it with no login at all. Try one URL: curl -X POST https://request.usestring.ai/v1/fetch -H "Authorization: Bearer $STRING_API_KEY" -d '{"url":"<page>"}'.

Step 2: Copy the session cookie for a quick job

Sign in with your browser, open the developer tools, and copy the request's Cookie header from the Network tab. Send it with your requests. This is the fastest way to pull a few pages, and the first answer most forums give.

It stops working the moment the session expires, and it has no way to sign in again. Treat it as a test, not a pipeline.

Step 3: Script the login form with a session

For a plain HTML form, post your credentials once and let a session carry the cookies. The Python requests docs say the Session object "persists cookies across all requests made from the Session instance" (requests docs).

Most forms carry a hidden CSRF token, so read the form first and send every hidden field back with your credentials:

import os

import requests
from bs4 import BeautifulSoup

LOGIN_URL = "https://example.com/login"
DATA_URL = "https://example.com/account/orders"

with requests.Session() as s:
    s.headers["User-Agent"] = (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36"
    )

    # 1. Load the form and keep every hidden field (CSRF token included).
    form = BeautifulSoup(s.get(LOGIN_URL, timeout=30).text, "html.parser").find("form")
    payload = {i["name"]: i.get("value", "") for i in form.find_all("input", type="hidden") if i.get("name")}
    payload.update({"email": os.environ["SITE_USER"], "password": os.environ["SITE_PASS"]})

    # 2. Post the form. The session stores the cookies the site sets.
    action = requests.compat.urljoin(LOGIN_URL, form.get("action") or LOGIN_URL)
    s.post(action, data=payload, timeout=30).raise_for_status()

    # 3. Every later request carries the login.
    page = s.get(DATA_URL, timeout=30)
    if "Sign in" in page.text or page.url.startswith(LOGIN_URL):
        raise SystemExit("Not signed in: check the field names and the CSRF token")
    print(page.text[:500])

Read the field names (email, password) from the form's HTML; they differ on every site. Keep credentials in environment variables, never in the script.

Step 4: Sign in with a real browser and save its state

When the login runs JavaScript, sets cookies from scripts, or shows a bot check, use a real browser. Playwright's docs describe the pattern: "log in only once and then skip the log in step" by saving the browser's storage state and loading it into new contexts (Playwright docs).

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)

    # Run once: sign in by hand (or script it), then save cookies and storage.
    context = browser.new_context()
    page = context.new_page()
    page.goto("https://example.com/login")
    input("Sign in in the browser window, then press Enter here...")
    context.storage_state(path="state.json")
    context.close()

    # Every later run: start already signed in.
    context = browser.new_context(storage_state="state.json")
    page = context.new_page()
    page.goto("https://example.com/account/orders")
    print(page.content()[:500])
    browser.close()

Playwright warns that the state file "may contain sensitive cookies and headers that could be used to impersonate you or your test account", so keep state.json out of every repository.

Step 5: Move the browser and the login to a hosted profile

At scale, the hard parts are hosting the browsers and keeping a login alive on a clean IP. String's browser contexts store the profile server-side. This is the Python version of the flow in String's docs: create a profile, sign in once, then reconnect signed in.

import os

import requests
from playwright.sync_api import sync_playwright

API = "https://request.usestring.ai/v1"
AUTH = {"Authorization": f"Bearer {os.environ['STRING_API_KEY']}"}

# 1. Create a profile and keep its id.
context_id = requests.post(f"{API}/contexts", headers=AUTH, json={"name": "orders-login"}, timeout=30).json()["id"]

def connect(p, persist=True):
    url = f"wss://request.usestring.ai/v1/wss?contextId={context_id}&persist={str(persist).lower()}&countryCode=US"
    return p.chromium.connect_over_cdp(url, headers=AUTH, timeout=90_000)

with sync_playwright() as p:
    # 2. Sign in once in the default context. Closing the browser saves the profile.
    browser = connect(p)
    context = browser.contexts[0]
    page = context.pages[0] if context.pages else context.new_page()
    page.goto("https://example.com/login")
    # ...fill in the login form and submit...
    browser.close()

    # 3. Reconnect read-only: the saved login is restored.
    browser = connect(p, persist=False)
    context = browser.contexts[0]
    page = context.pages[0] if context.pages else context.new_page()
    page.goto("https://example.com/account/orders")
    print(page.content()[:500])
    browser.close()

Use browser.contexts[0], not browser.new_context(): only the default context is saved. Send the same countryCode every time, because a login restored from another country may be challenged or signed out.

The failure mode: an expired session that returns HTTP 200

The costly failure is the one your code does not see. When a session expires, most sites answer with a redirect to the login page or a 200 page that says "Sign in", not an error. A scraper that checks the status code stores the login page as data, run after run.

Check every response for something only a signed-in page has: your account name, an order ID, a "Sign out" link. When the check fails, sign in again and retry once, then stop and alert. Our answer on detecting a block behind HTTP 200 covers content checks, fixing 403 errors covers the refusals that come before a login works, and scraping without getting blocked covers the headers, fingerprints and pace that keep a signed-in scraper running.

Read the terms before you sign in

A login is where the site's terms start to bind you. In Van Buren v. United States (2021), the US Supreme Court read the federal anti-hacking law as a "gates-up-or-down inquiry": you either can or cannot access an area of a system (opinion). Your own login opens your own gates, and no one else's.

Contract claims are a separate risk. In 2022 a federal court found that hiQ "breached LinkedIn's User Agreement both through its own scraping of LinkedIn's site and using scraped data, and through turkers' creation of false identities" (Proskauer summary). In 2024 the same court found "a strong and compelling argument" that Bright Data was not bound by Meta's terms when it "scraped public data while not logged-in" (Eric Goldman blog). The contrast is the point: signed-out scraping of public pages and signed-in scraping sit on different legal ground. Our pages on the hiQ v. LinkedIn ruling, whether web scraping is legal and personal data under GDPR and CCPA go further. None of this is legal advice.

The honest counterpoint

Sometimes the right answer is not to scrape. Asked how to avoid bans on a login-based site, the top Stack Overflow answer said there is no way, and pointed elsewhere:

"However, the most effective way would be to contact the website owner and offer a partnership or pay for accessing the data using an API or something like that if they have this service."

Paulo Marques, Stack Overflow, March 2021

The asker had already seen why: every request made with one login can be tied back to that account, whatever IP it comes from. If the data is behind someone else's login, ask for it. If it is behind yours, Step 3 or Step 4 will usually do. To test the hosted path on your own login, start with a free key and the Step 5 script.

FAQ

How to scrape a website that requires login?

Sign in once, keep the session cookies the site sets, and send them with every later request. For a plain HTML form, post your credentials with Python requests.Session and the form's hidden CSRF field. For a JavaScript login or a bot check, sign in with Playwright and save its storage_state. Use only accounts you own, on sites whose terms allow it.

How do I scrape a website that requires login with Python?

Use requests.Session(): load the login page, collect the form's hidden fields, add your username and password, and post the form. The session keeps the cookies, so later s.get() calls are signed in. If the login needs JavaScript, switch to Playwright for Python and reuse a saved storage_state file.

Can I scrape a site behind login with BeautifulSoup?

BeautifulSoup only parses HTML; it does not sign in. Pair it with requests.Session to handle the login and cookies, then pass each page's HTML to BeautifulSoup. Use it to read the login form's hidden fields too.

How do I keep a login session alive while scraping?

Save the session and check every response for a signed-in marker, such as your account name or a "Sign out" link. When the marker is missing, sign in again and retry once. Saved browser state from Playwright or a hosted profile restores cookies and site storage, but the site's own rules can still expire it.

How do I scrape a site that uses two-factor authentication?

Sign in once by hand in a real browser, complete the two-factor step, and save the browser state. Later runs reuse the saved session until it expires. Playwright's storage_state and String's browser contexts both work this way, so you enter the code only when the site asks again.

Is it legal to scrape a website that requires login?

It depends on whose account it is and what the site's terms say. Logging in usually means accepting the site's terms, and in 2022 a US court found hiQ breached LinkedIn's terms, partly through logged-in fake accounts. Scraping your own account data is a different case from scraping behind someone else's login. Ask a lawyer for your case.

Will a website ban my account for scraping?

It can. Every request made with one login is tied to that account, so rotating IPs does not hide it. Keep your pace close to a human's, scrape only what your account may see, and prefer an official API or export when the site offers one.

Can a scraping API log in for me?

Many scraping APIs take one URL per request and leave the login to you. String offers a remote browser with saved profiles: you sign in once over Playwright or Puppeteer, and later sessions start signed in, with CAPTCHA solving available in the session. Sign-in paths on some sites need approval first.

Sources

• Stack Overflow, How to scrape a website which requires login using Python and BeautifulSoup (answers by Adelin, April 2020, and Plabon Dutta, February 2020).
• Stack Overflow, How to avoid being banned while scraping data from a login-based site (answer by Paulo Marques, March 2021).
• Requests docs, Advanced usage: Session objects.
• Playwright for Python, Authentication.
• US Supreme Court, Van Buren v. United States, 593 U.S. 374 (2021), via Cornell LII.
• Proskauer, Court finds hiQ breached LinkedIn's terms prohibiting scraping, November 2022.
• Eric Goldman blog, Bright Data scores major victory in web scraping dispute with Meta, January 2024, and Facebook drops anti-scraping lawsuit against Bright Data, February 2024.
• String docs, Browser contexts, Browser overview, Captcha solving in browser sessions, Access control and Fetch overview, read November 4, 2026.
• String pricing, rates confirmed October 22, 2026, checked November 4, 2026.
• Web Data Frontier Benchmark, October 30, 2026 run (write-up).