src.subpages.attention
A group of neurons tend to fire in response to commas and other punctuation. Other groups of neurons tend to fire in response to pronouns. Use this visualization to factorize neuron activity in individual FFNN layers or in the entire model.
1""" 2A group of neurons tend to fire in response to commas and other punctuation. Other groups of neurons tend to fire in response to pronouns. Use this visualization to factorize neuron activity in individual FFNN layers or in the entire model. 3""" 4import ecco 5import streamlit as st 6from streamlit.components.v1 import html 7 8from src.subpages.page import Context, Page # type: ignore 9 10_SETUP_HTML = """ 11<script src="https://requirejs.org/docs/release/2.3.6/minified/require.js"></script> 12<script> 13 var ecco_url = 'https://storage.googleapis.com/ml-intro/ecco/' 14 //var ecco_url = 'http://localhost:8000/' 15 16 if (window.ecco === undefined) window.ecco = {} 17 18 // Setup the paths of the script we'll be using 19 requirejs.config({ 20 urlArgs: "bust=" + (new Date()).getTime(), 21 nodeRequire: require, 22 paths: { 23 d3: "https://d3js.org/d3.v6.min", // This is only for use in setup.html and basic.html 24 "d3-array": "https://d3js.org/d3-array.v2.min", 25 jquery: "https://code.jquery.com/jquery-3.5.1.min", 26 ecco: ecco_url + 'js/0.0.6/ecco-bundle.min', 27 xregexp: 'https://cdnjs.cloudflare.com/ajax/libs/xregexp/3.2.0/xregexp-all.min' 28 } 29 }); 30 31 // Add the css file 32 //requirejs(['d3'], 33 // function (d3) { 34 // d3.select('#css').attr('href', ecco_url + 'html/styles.css') 35 // }) 36 37 console.log('Ecco initialize!!') 38 39 // returns a 'basic' object. basic.init() selects the html div we'll be 40 // rendering the html into, adds styles.css to the document. 41 define('basic', ['d3'], 42 function (d3) { 43 return { 44 init: function (viz_id = null) { 45 if (viz_id == null) { 46 viz_id = "viz_" + Math.round(Math.random() * 10000000) 47 } 48 // Select the div rendered below, change its id 49 const div = d3.select('#basic').attr('id', viz_id), 50 div_parent = d3.select('#' + viz_id).node().parentNode 51 52 // Link to CSS file 53 d3.select(div_parent).insert('link') 54 .attr('rel', 'stylesheet') 55 .attr('type', 'text/css') 56 .attr('href', ecco_url + 'html/0.0.2/styles.css') 57 58 return viz_id 59 } 60 } 61 }, function (err) { 62 console.log(err); 63 } 64 ) 65</script> 66 67<head> 68 <link id='css' rel="stylesheet" type="text/css"> 69</head> 70<div id="basic"></div> 71""" 72 73 74@st.cache(allow_output_mutation=True) 75def _load_ecco_model(): 76 model_config = { 77 "embedding": "embeddings.word_embeddings", 78 "type": "mlm", 79 "activations": [r"ffn\.lin1"], 80 "token_prefix": "", 81 "partial_token_prefix": "##", 82 } 83 return ecco.from_pretrained( 84 "elastic/distilbert-base-uncased-finetuned-conll03-english", 85 model_config=model_config, 86 activations=True, 87 ) 88 89 90class AttentionPage(Page): 91 name = "Activations" 92 icon = "activity" 93 94 def get_widget_defaults(self): 95 return { 96 "act_n_components": 8, 97 "act_default_text": """Now I ask you: what can be expected of man since he is a being endowed with strange qualities? Shower upon him every earthly blessing, drown him in a sea of happiness, so that nothing but bubbles of bliss can be seen on the surface; give him economic prosperity, such that he should have nothing else to do but sleep, eat cakes and busy himself with the continuation of his species, and even then out of sheer ingratitude, sheer spite, man would play you some nasty trick. He would even risk his cakes and would deliberately desire the most fatal rubbish, the most uneconomical absurdity, simply to introduce into all this positive good sense his fatal fantastic element. It is just his fantastic dreams, his vulgar folly that he will desire to retain, simply in order to prove to himself--as though that were so necessary-- that men still are men and not the keys of a piano, which the laws of nature threaten to control so completely that soon one will be able to desire nothing but by the calendar. And that is not all: even if man really were nothing but a piano-key, even if this were proved to him by natural science and mathematics, even then he would not become reasonable, but would purposely do something perverse out of simple ingratitude, simply to gain his point. And if he does not find means he will contrive destruction and chaos, will contrive sufferings of all sorts, only to gain his point! He will launch a curse upon the world, and as only man can curse (it is his privilege, the primary distinction between him and other animals), may be by his curse alone he will attain his object--that is, convince himself that he is a man and not a piano-key!""", 98 "act_from_layer": 0, 99 "act_to_layer": 6, 100 } 101 102 def render(self, context: Context): 103 st.title(self.name) 104 105 with st.expander("ℹ️", expanded=True): 106 st.write( 107 "A group of neurons tend to fire in response to commas and other punctuation. Other groups of neurons tend to fire in response to pronouns. Use this visualization to factorize neuron activity in individual FFNN layers or in the entire model." 108 ) 109 110 lm = _load_ecco_model() 111 112 col1, _, col2 = st.columns([1.5, 0.5, 4]) 113 with col1: 114 st.subheader("Settings") 115 n_components = st.slider( 116 "#components", 117 key="act_n_components", 118 min_value=2, 119 max_value=10, 120 step=1, 121 ) 122 from_layer = ( 123 st.slider( 124 "from layer", 125 key="act_from_layer", 126 value=0, 127 min_value=0, 128 max_value=len(lm.model.transformer.layer) - 1, 129 step=1, 130 ) 131 or None 132 ) 133 to_layer = ( 134 st.slider( 135 "to layer", 136 key="act_to_layer", 137 value=0, 138 min_value=0, 139 max_value=len(lm.model.transformer.layer), 140 step=1, 141 ) 142 or None 143 ) 144 with col2: 145 st.subheader("–") 146 text = st.text_area("Text", key="act_default_text", height=240) 147 148 inputs = lm.tokenizer([text], return_tensors="pt") 149 output = lm(inputs) 150 nmf = output.run_nmf(n_components=n_components, from_layer=from_layer, to_layer=to_layer) 151 data = nmf.explore(returnData=True) 152 _JS_TEMPLATE = f"""<script>requirejs(['basic', 'ecco'], function(basic, ecco){{ 153 const viz_id = basic.init() 154 ecco.interactiveTokensAndFactorSparklines(viz_id, {data}, {{ 'hltrCFG': {{'tokenization_config': {{'token_prefix': '', 'partial_token_prefix': '##'}} }} }}) 155 }}, function (err) {{ 156 console.log(err); 157 }})</script>""" 158 html(_SETUP_HTML + _JS_TEMPLATE, height=800, scrolling=True)
91class AttentionPage(Page): 92 name = "Activations" 93 icon = "activity" 94 95 def get_widget_defaults(self): 96 return { 97 "act_n_components": 8, 98 "act_default_text": """Now I ask you: what can be expected of man since he is a being endowed with strange qualities? Shower upon him every earthly blessing, drown him in a sea of happiness, so that nothing but bubbles of bliss can be seen on the surface; give him economic prosperity, such that he should have nothing else to do but sleep, eat cakes and busy himself with the continuation of his species, and even then out of sheer ingratitude, sheer spite, man would play you some nasty trick. He would even risk his cakes and would deliberately desire the most fatal rubbish, the most uneconomical absurdity, simply to introduce into all this positive good sense his fatal fantastic element. It is just his fantastic dreams, his vulgar folly that he will desire to retain, simply in order to prove to himself--as though that were so necessary-- that men still are men and not the keys of a piano, which the laws of nature threaten to control so completely that soon one will be able to desire nothing but by the calendar. And that is not all: even if man really were nothing but a piano-key, even if this were proved to him by natural science and mathematics, even then he would not become reasonable, but would purposely do something perverse out of simple ingratitude, simply to gain his point. And if he does not find means he will contrive destruction and chaos, will contrive sufferings of all sorts, only to gain his point! He will launch a curse upon the world, and as only man can curse (it is his privilege, the primary distinction between him and other animals), may be by his curse alone he will attain his object--that is, convince himself that he is a man and not a piano-key!""", 99 "act_from_layer": 0, 100 "act_to_layer": 6, 101 } 102 103 def render(self, context: Context): 104 st.title(self.name) 105 106 with st.expander("ℹ️", expanded=True): 107 st.write( 108 "A group of neurons tend to fire in response to commas and other punctuation. Other groups of neurons tend to fire in response to pronouns. Use this visualization to factorize neuron activity in individual FFNN layers or in the entire model." 109 ) 110 111 lm = _load_ecco_model() 112 113 col1, _, col2 = st.columns([1.5, 0.5, 4]) 114 with col1: 115 st.subheader("Settings") 116 n_components = st.slider( 117 "#components", 118 key="act_n_components", 119 min_value=2, 120 max_value=10, 121 step=1, 122 ) 123 from_layer = ( 124 st.slider( 125 "from layer", 126 key="act_from_layer", 127 value=0, 128 min_value=0, 129 max_value=len(lm.model.transformer.layer) - 1, 130 step=1, 131 ) 132 or None 133 ) 134 to_layer = ( 135 st.slider( 136 "to layer", 137 key="act_to_layer", 138 value=0, 139 min_value=0, 140 max_value=len(lm.model.transformer.layer), 141 step=1, 142 ) 143 or None 144 ) 145 with col2: 146 st.subheader("–") 147 text = st.text_area("Text", key="act_default_text", height=240) 148 149 inputs = lm.tokenizer([text], return_tensors="pt") 150 output = lm(inputs) 151 nmf = output.run_nmf(n_components=n_components, from_layer=from_layer, to_layer=to_layer) 152 data = nmf.explore(returnData=True) 153 _JS_TEMPLATE = f"""<script>requirejs(['basic', 'ecco'], function(basic, ecco){{ 154 const viz_id = basic.init() 155 ecco.interactiveTokensAndFactorSparklines(viz_id, {data}, {{ 'hltrCFG': {{'tokenization_config': {{'token_prefix': '', 'partial_token_prefix': '##'}} }} }}) 156 }}, function (err) {{ 157 console.log(err); 158 }})</script>""" 159 html(_SETUP_HTML + _JS_TEMPLATE, height=800, scrolling=True)
Base class for all pages.
def
get_widget_defaults(self)
95 def get_widget_defaults(self): 96 return { 97 "act_n_components": 8, 98 "act_default_text": """Now I ask you: what can be expected of man since he is a being endowed with strange qualities? Shower upon him every earthly blessing, drown him in a sea of happiness, so that nothing but bubbles of bliss can be seen on the surface; give him economic prosperity, such that he should have nothing else to do but sleep, eat cakes and busy himself with the continuation of his species, and even then out of sheer ingratitude, sheer spite, man would play you some nasty trick. He would even risk his cakes and would deliberately desire the most fatal rubbish, the most uneconomical absurdity, simply to introduce into all this positive good sense his fatal fantastic element. It is just his fantastic dreams, his vulgar folly that he will desire to retain, simply in order to prove to himself--as though that were so necessary-- that men still are men and not the keys of a piano, which the laws of nature threaten to control so completely that soon one will be able to desire nothing but by the calendar. And that is not all: even if man really were nothing but a piano-key, even if this were proved to him by natural science and mathematics, even then he would not become reasonable, but would purposely do something perverse out of simple ingratitude, simply to gain his point. And if he does not find means he will contrive destruction and chaos, will contrive sufferings of all sorts, only to gain his point! He will launch a curse upon the world, and as only man can curse (it is his privilege, the primary distinction between him and other animals), may be by his curse alone he will attain his object--that is, convince himself that he is a man and not a piano-key!""", 99 "act_from_layer": 0, 100 "act_to_layer": 6, 101 }
This function holds the default settings for all the page's widgets.
Returns
dict: A dictionary of widget defaults, where the keys are the widget names and the values are the default.
103 def render(self, context: Context): 104 st.title(self.name) 105 106 with st.expander("ℹ️", expanded=True): 107 st.write( 108 "A group of neurons tend to fire in response to commas and other punctuation. Other groups of neurons tend to fire in response to pronouns. Use this visualization to factorize neuron activity in individual FFNN layers or in the entire model." 109 ) 110 111 lm = _load_ecco_model() 112 113 col1, _, col2 = st.columns([1.5, 0.5, 4]) 114 with col1: 115 st.subheader("Settings") 116 n_components = st.slider( 117 "#components", 118 key="act_n_components", 119 min_value=2, 120 max_value=10, 121 step=1, 122 ) 123 from_layer = ( 124 st.slider( 125 "from layer", 126 key="act_from_layer", 127 value=0, 128 min_value=0, 129 max_value=len(lm.model.transformer.layer) - 1, 130 step=1, 131 ) 132 or None 133 ) 134 to_layer = ( 135 st.slider( 136 "to layer", 137 key="act_to_layer", 138 value=0, 139 min_value=0, 140 max_value=len(lm.model.transformer.layer), 141 step=1, 142 ) 143 or None 144 ) 145 with col2: 146 st.subheader("–") 147 text = st.text_area("Text", key="act_default_text", height=240) 148 149 inputs = lm.tokenizer([text], return_tensors="pt") 150 output = lm(inputs) 151 nmf = output.run_nmf(n_components=n_components, from_layer=from_layer, to_layer=to_layer) 152 data = nmf.explore(returnData=True) 153 _JS_TEMPLATE = f"""<script>requirejs(['basic', 'ecco'], function(basic, ecco){{ 154 const viz_id = basic.init() 155 ecco.interactiveTokensAndFactorSparklines(viz_id, {data}, {{ 'hltrCFG': {{'tokenization_config': {{'token_prefix': '', 'partial_token_prefix': '##'}} }} }}) 156 }}, function (err) {{ 157 console.log(err); 158 }})</script>""" 159 html(_SETUP_HTML + _JS_TEMPLATE, height=800, scrolling=True)
This function renders the page.